[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:39 dist-test-slave-lbpt krb5kdc[12343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:39 dist-test-slave-lbpt krb5kdc[12343](info): set up 2 sockets
Jun 21 07:51:39 dist-test-slave-lbpt krb5kdc[12343](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260621 07:51:40.126544 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:51:40 dist-test-slave-lbpt krb5kdc[12343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:51:40.245805 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:42509
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:42509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:40.355085 12356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:40.355484 12356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:40.355564 12356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:40.359568 12356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:40.359660 12356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:40.359699 12356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:40.359736 12356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:40.359772 12356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:40.365036 12356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:42509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:42509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:40.366160 12356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:40.367219 12356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:40.375577 12361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:40.375594 12364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:40.375638 12362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:40.375948 12356 server_base.cc:1034] running on GCE node
I20260621 07:51:40.376565 12356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:40.377718 12356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:40.378899 12356 hybrid_clock.cc:648] HybridClock initialized: now 1782028300378884 us; error 31 us; skew 500 ppm
Jun 21 07:51:40 dist-test-slave-lbpt krb5kdc[12343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028300, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:40.381673 12356 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:51:40.382618 12356 webserver.cc:460] Webserver started at http://127.12.12.62:39415/ using document root <none> and password file <none>
I20260621 07:51:40.383188 12356 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:40.383276 12356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:40.383507 12356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:40.385579 12356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "c8a28cae00e34543b42283098abcde0c"
format_stamp: "Formatted at 2026-06-21 07:51:40 on dist-test-slave-lbpt"
server_key: "ea90845f5f362cb8dca3dddaf04deabe"
server_key_iv: "a4bce23ee8aa6d1a23ea0791d7883cc7"
server_key_version: "encryptionkey@0"
I20260621 07:51:40.386170 12356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "c8a28cae00e34543b42283098abcde0c"
format_stamp: "Formatted at 2026-06-21 07:51:40 on dist-test-slave-lbpt"
server_key: "ea90845f5f362cb8dca3dddaf04deabe"
server_key_iv: "a4bce23ee8aa6d1a23ea0791d7883cc7"
server_key_version: "encryptionkey@0"
I20260621 07:51:40.389998 12356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:51:40.392552 12371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:40.393894 12356 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:40.394029 12356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "c8a28cae00e34543b42283098abcde0c"
format_stamp: "Formatted at 2026-06-21 07:51:40 on dist-test-slave-lbpt"
server_key: "ea90845f5f362cb8dca3dddaf04deabe"
server_key_iv: "a4bce23ee8aa6d1a23ea0791d7883cc7"
server_key_version: "encryptionkey@0"
I20260621 07:51:40.394145 12356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:40.413794 12356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:40.417320 12356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:40.417536 12356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:40.424916 12356 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:42509
I20260621 07:51:40.424947 12423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:42509 every 8 connection(s)
I20260621 07:51:40.425930 12356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:40.429060 12424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:40.432554 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 12356
I20260621 07:51:40.432720 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:51:40.434363 12336 external_mini_cluster.cc:1428] Setting key c0baae75751c0692f689f7f0da67c094
I20260621 07:51:40.435788 12424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c: Bootstrap starting.
I20260621 07:51:40.438663 12424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:40.439582 12424 log.cc:824] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:40.441867 12424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c: No bootstrap required, opened a new log
I20260621 07:51:40.446947 12424 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a28cae00e34543b42283098abcde0c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42509 } }
I20260621 07:51:40.447235 12424 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:40.447300 12424 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a28cae00e34543b42283098abcde0c, State: Initialized, Role: FOLLOWER
I20260621 07:51:40.447813 12424 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a28cae00e34543b42283098abcde0c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42509 } }
I20260621 07:51:40.447978 12424 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:40.448065 12424 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:40.448199 12424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:40.449154 12424 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a28cae00e34543b42283098abcde0c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42509 } }
I20260621 07:51:40.449527 12424 leader_election.cc:302] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8a28cae00e34543b42283098abcde0c; no voters:
I20260621 07:51:40.449947 12424 leader_election.cc:288] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:40.450455 12430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:40.450726 12430 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [term 1 LEADER]: Becoming Leader. State: Replica: c8a28cae00e34543b42283098abcde0c, State: Running, Role: LEADER
I20260621 07:51:40.451164 12430 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a28cae00e34543b42283098abcde0c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42509 } }
I20260621 07:51:40.451455 12424 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:40.452981 12429 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8a28cae00e34543b42283098abcde0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a28cae00e34543b42283098abcde0c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42509 } } }
I20260621 07:51:40.452989 12430 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8a28cae00e34543b42283098abcde0c. Latest consensus state: current_term: 1 leader_uuid: "c8a28cae00e34543b42283098abcde0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8a28cae00e34543b42283098abcde0c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42509 } } }
I20260621 07:51:40.453133 12429 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:40.453137 12430 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:40.453534 12441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 21 07:51:40 dist-test-slave-lbpt krb5kdc[12343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:40.456907 12441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:40.463475 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:40.438068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.12.12.62:42509)
0621 07:51:40.438589 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:40.438609 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:40.439644 (+ 1035us) server_negotiation.cc:407] Connection header received
0621 07:51:40.444383 (+ 4739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:40.444401 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:40.444718 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:40.444989 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:40.448145 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:40.449199 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:40.451384 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:40.452161 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:40.452354 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:40.457685 (+ 5331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:40.457690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:40.457703 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:40.457731 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:40.459556 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:40.460239 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:40.460248 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:40.460255 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:40.460407 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:40.460811 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:40.460814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:40.460817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:40.461369 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:40.461581 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:40.462230 (+ 649us) server_negotiation.cc:299] Negotiation successful
0621 07:51:40.462529 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":155,"threads_started":1}
I20260621 07:51:40.465101 12441 catalog_manager.cc:1347] Generated new cluster ID: 3e32fb046cae47408b4dc621fdbf1c5e
I20260621 07:51:40.465173 12441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:40.484987 12441 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:40.485997 12441 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:40.490527 12441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8a28cae00e34543b42283098abcde0c: Generated new TSK 0
I20260621 07:51:40.491261 12441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:51:40.525080 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:42509
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:40.649977 12450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:40.650243 12450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:40.650286 12450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:40.654474 12450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:40.654531 12450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:40.654588 12450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:51:40.659778 12450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:42509
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:40.660737 12450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:40.661681 12450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:40.671213 12455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:40.671259 12456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:40.671259 12458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:40.671427 12450 server_base.cc:1034] running on GCE node
I20260621 07:51:40.671870 12450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:40.672487 12450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:40.673643 12450 hybrid_clock.cc:648] HybridClock initialized: now 1782028300673624 us; error 33 us; skew 500 ppm
Jun 21 07:51:40 dist-test-slave-lbpt krb5kdc[12343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028300, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:40.676360 12450 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:51:40.677148 12450 webserver.cc:460] Webserver started at http://127.12.12.1:40043/ using document root <none> and password file <none>
I20260621 07:51:40.677670 12450 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:40.677726 12450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:40.677894 12450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:40.679884 12450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "0078bde27860431d86af3c1f43e43ca4"
format_stamp: "Formatted at 2026-06-21 07:51:40 on dist-test-slave-lbpt"
server_key: "0d0904466eb7e37c1bdf9b255d385b6c"
server_key_iv: "4fe3c78c7a57ee685f2f921da9ee97f4"
server_key_version: "encryptionkey@0"
I20260621 07:51:40.680399 12450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "0078bde27860431d86af3c1f43e43ca4"
format_stamp: "Formatted at 2026-06-21 07:51:40 on dist-test-slave-lbpt"
server_key: "0d0904466eb7e37c1bdf9b255d385b6c"
server_key_iv: "4fe3c78c7a57ee685f2f921da9ee97f4"
server_key_version: "encryptionkey@0"
I20260621 07:51:40.683931 12450 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260621 07:51:40.686268 12465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:40.687508 12450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 07:51:40.687626 12450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "0078bde27860431d86af3c1f43e43ca4"
format_stamp: "Formatted at 2026-06-21 07:51:40 on dist-test-slave-lbpt"
server_key: "0d0904466eb7e37c1bdf9b255d385b6c"
server_key_iv: "4fe3c78c7a57ee685f2f921da9ee97f4"
server_key_version: "encryptionkey@0"
I20260621 07:51:40.687732 12450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:40.704145 12450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:40.707551 12450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:40.707752 12450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:40.708359 12450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:40.709182 12450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:40.709228 12450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:40.709288 12450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:40.709327 12450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:40.717969 12450 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:36919
I20260621 07:51:40.718031 12578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:36919 every 8 connection(s)
I20260621 07:51:40.718930 12450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:51:40.722234 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 12450
I20260621 07:51:40.722337 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:51:40.722605 12336 external_mini_cluster.cc:1428] Setting key 27232e6c449dc95631f5b10f77127146
Jun 21 07:51:40 dist-test-slave-lbpt krb5kdc[12343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028300, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
I20260621 07:51:40.733980 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:40.720704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:37773 (local address 127.12.12.62:42509)
0621 07:51:40.720872 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:40.720877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:40.721677 (+ 800us) server_negotiation.cc:407] Connection header received
0621 07:51:40.723132 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:40.723135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:40.723202 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:40.723295 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:40.724709 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:40.725582 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:40.726935 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:40.727338 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:40.727473 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:40.729994 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:40.730000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:40.730005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:40.730044 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:40.731810 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:40.732346 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:40.732352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:40.732356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:40.732452 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:40.732779 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:40.732785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:40.732787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:40.733083 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:40.733193 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:40.733673 (+ 480us) server_negotiation.cc:299] Negotiation successful
0621 07:51:40.733780 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 07:51:40.734925 12581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:40.721092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:42509 (local address 127.12.12.1:37773)
0621 07:51:40.721518 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:40.721563 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:40.722890 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:40.723493 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:40.723503 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:40.724008 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:40.724562 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:40.724576 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:40.725773 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:40.725776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:40.726795 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:40.726805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:40.727560 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:40.727565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:40.727723 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:40.728317 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:40.728334 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:40.729818 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:40.731933 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:40.731937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:40.731951 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:40.732230 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:40.732541 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:40.732543 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:40.732545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:40.732678 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:40.733199 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:40.733204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:40.733452 (+ 248us) client_negotiation.cc:769] Sending connection context
0621 07:51:40.733627 (+ 175us) client_negotiation.cc:240] Negotiation successful
0621 07:51:40.733982 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
I20260621 07:51:40.736253 12579 heartbeater.cc:347] Connected to a master server at 127.12.12.62:42509
I20260621 07:51:40.736508 12579 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:40.736992 12579 heartbeater.cc:511] Master 127.12.12.62:42509 requested a full tablet report, sending...
I20260621 07:51:40.738543 12388 ts_manager.cc:194] Registered new tserver with Master: 0078bde27860431d86af3c1f43e43ca4 (127.12.12.1:36919)
I20260621 07:51:40.739693 12388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:37773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:51:40.750156 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:42509
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:40.863548 12584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:40.863816 12584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:40.863860 12584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:40.868168 12584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:40.868225 12584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:40.868286 12584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:51:40.873428 12584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:42509
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:40.874464 12584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:40.875769 12584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:40.885972 12592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:40.886070 12589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:40.885987 12590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:40.886234 12584 server_base.cc:1034] running on GCE node
I20260621 07:51:40.886713 12584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:40.887364 12584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:40.888547 12584 hybrid_clock.cc:648] HybridClock initialized: now 1782028300888533 us; error 24 us; skew 500 ppm
Jun 21 07:51:40 dist-test-slave-lbpt krb5kdc[12343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028300, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:40.891459 12584 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:51:40.892411 12584 webserver.cc:460] Webserver started at http://127.12.12.2:46333/ using document root <none> and password file <none>
I20260621 07:51:40.892940 12584 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:40.892997 12584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:40.893162 12584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:40.895216 12584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "e9f6f951ebbd4d80a20211c1bcbc93bb"
format_stamp: "Formatted at 2026-06-21 07:51:40 on dist-test-slave-lbpt"
server_key: "75581cfdbaaba5555785e78f4a0a7cc0"
server_key_iv: "c0a7afba9f77d9ba6a743d342f9570a8"
server_key_version: "encryptionkey@0"
I20260621 07:51:40.895714 12584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "e9f6f951ebbd4d80a20211c1bcbc93bb"
format_stamp: "Formatted at 2026-06-21 07:51:40 on dist-test-slave-lbpt"
server_key: "75581cfdbaaba5555785e78f4a0a7cc0"
server_key_iv: "c0a7afba9f77d9ba6a743d342f9570a8"
server_key_version: "encryptionkey@0"
I20260621 07:51:40.899526 12584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:51:40.901894 12599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:40.903276 12584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:51:40.903373 12584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "e9f6f951ebbd4d80a20211c1bcbc93bb"
format_stamp: "Formatted at 2026-06-21 07:51:40 on dist-test-slave-lbpt"
server_key: "75581cfdbaaba5555785e78f4a0a7cc0"
server_key_iv: "c0a7afba9f77d9ba6a743d342f9570a8"
server_key_version: "encryptionkey@0"
I20260621 07:51:40.903468 12584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:40.934295 12584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:40.937238 12584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:40.937407 12584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:40.937944 12584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:40.938756 12584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:40.938802 12584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:40.938843 12584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:40.938863 12584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:40.947391 12584 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:34735
I20260621 07:51:40.947417 12712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:34735 every 8 connection(s)
I20260621 07:51:40.948287 12584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:51:40.956985 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 12584
I20260621 07:51:40.957104 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:51:40.957412 12336 external_mini_cluster.cc:1428] Setting key 5f7236d790818f7f7dafcda5602056ea
Jun 21 07:51:40 dist-test-slave-lbpt krb5kdc[12343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028300, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:40.962461 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:40.950081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:42781 (local address 127.12.12.62:42509)
0621 07:51:40.950227 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:40.950231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:40.951276 (+ 1045us) server_negotiation.cc:407] Connection header received
0621 07:51:40.952229 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:40.952233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:40.952290 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:40.952390 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:40.953714 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:40.954376 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:40.955560 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:40.955808 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:40.955904 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:40.958743 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:40.958747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:40.958750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:40.958776 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:40.960281 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:40.960944 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:40.960949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:40.960951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:40.961032 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:40.961364 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:40.961369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:40.961371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:40.961551 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:40.961650 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:40.962158 (+ 508us) server_negotiation.cc:299] Negotiation successful
0621 07:51:40.962278 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 07:51:40.963447 12715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:40.950506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:42509 (local address 127.12.12.2:42781)
0621 07:51:40.951132 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:40.951173 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:40.952032 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:40.952580 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:40.952587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:40.952987 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:40.953506 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:40.953529 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:40.954539 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:40.954541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:40.955435 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:40.955444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:40.955948 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:40.955951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:40.956092 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:40.956757 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:40.956781 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:40.958593 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:40.960394 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:40.960403 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:40.960424 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:40.960833 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:40.961128 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:40.961131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:40.961133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:40.961272 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:40.961662 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:40.961671 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:40.961920 (+ 249us) client_negotiation.cc:769] Sending connection context
0621 07:51:40.962104 (+ 184us) client_negotiation.cc:240] Negotiation successful
0621 07:51:40.962468 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":98,"threads_started":1}
I20260621 07:51:40.964915 12713 heartbeater.cc:347] Connected to a master server at 127.12.12.62:42509
I20260621 07:51:40.965152 12713 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:40.965631 12713 heartbeater.cc:511] Master 127.12.12.62:42509 requested a full tablet report, sending...
I20260621 07:51:40.966786 12388 ts_manager.cc:194] Registered new tserver with Master: e9f6f951ebbd4d80a20211c1bcbc93bb (127.12.12.2:34735)
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
I20260621 07:51:40.967587 12388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:42781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:51:40.983513 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:42509
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:41.098197 12718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:41.098552 12718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:41.098627 12718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:41.102836 12718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:41.102905 12718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:41.102993 12718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:51:41.108573 12718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:42509
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:41.109681 12718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:41.110801 12718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:41.120347 12723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:41.120491 12718 server_base.cc:1034] running on GCE node
W20260621 07:51:41.120334 12724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:41.120338 12726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:41.120990 12718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:41.121632 12718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:41.122809 12718 hybrid_clock.cc:648] HybridClock initialized: now 1782028301122790 us; error 29 us; skew 500 ppm
Jun 21 07:51:41 dist-test-slave-lbpt krb5kdc[12343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028301, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:41.125451 12718 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:51:41.126295 12718 webserver.cc:460] Webserver started at http://127.12.12.3:38679/ using document root <none> and password file <none>
I20260621 07:51:41.126914 12718 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:41.127002 12718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:41.127231 12718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:41.129405 12718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "2eb6a908f86e4071a031b0522be36836"
format_stamp: "Formatted at 2026-06-21 07:51:41 on dist-test-slave-lbpt"
server_key: "3840c44241b4334c9265fe7cb847f0dd"
server_key_iv: "fd8e1103c834873a08962c2fcfd9c459"
server_key_version: "encryptionkey@0"
I20260621 07:51:41.130015 12718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "2eb6a908f86e4071a031b0522be36836"
format_stamp: "Formatted at 2026-06-21 07:51:41 on dist-test-slave-lbpt"
server_key: "3840c44241b4334c9265fe7cb847f0dd"
server_key_iv: "fd8e1103c834873a08962c2fcfd9c459"
server_key_version: "encryptionkey@0"
I20260621 07:51:41.133883 12718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 07:51:41.136767 12733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:41.138031 12718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:41.138166 12718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "2eb6a908f86e4071a031b0522be36836"
format_stamp: "Formatted at 2026-06-21 07:51:41 on dist-test-slave-lbpt"
server_key: "3840c44241b4334c9265fe7cb847f0dd"
server_key_iv: "fd8e1103c834873a08962c2fcfd9c459"
server_key_version: "encryptionkey@0"
I20260621 07:51:41.138283 12718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:41.154820 12718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:41.158031 12718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:41.158236 12718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:41.158860 12718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:41.159662 12718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:41.159739 12718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:41.159806 12718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:41.159852 12718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:41.168570 12718 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:43697
I20260621 07:51:41.168596 12846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:43697 every 8 connection(s)
I20260621 07:51:41.169431 12718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:51:41.170382 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 12718
I20260621 07:51:41.170521 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:51:41.170856 12336 external_mini_cluster.cc:1428] Setting key 126aee686b9e1966b84fd456926ddaf7
Jun 21 07:51:41 dist-test-slave-lbpt krb5kdc[12343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028301, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:41.182966 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:41.171426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:45929 (local address 127.12.12.62:42509)
0621 07:51:41.171561 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:41.171565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:41.172509 (+ 944us) server_negotiation.cc:407] Connection header received
0621 07:51:41.173821 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:41.173823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:41.173867 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:41.173936 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:41.175177 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:41.175743 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:41.176835 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:41.177045 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:41.177121 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:41.179588 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:41.179591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:41.179593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:41.179621 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:41.181159 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:41.181612 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:41.181616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:41.181618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:41.181698 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:41.181983 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:41.181985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:41.181986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:41.182175 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:41.182252 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:41.182747 (+ 495us) server_negotiation.cc:299] Negotiation successful
0621 07:51:41.182845 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:51:41.183847 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:41.171920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:42509 (local address 127.12.12.3:45929)
0621 07:51:41.172375 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:41.172426 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:41.173624 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:41.174095 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:41.174103 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:41.174518 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:41.175057 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:41.175070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:41.175857 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:41.175860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:41.176734 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:41.176743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:41.177163 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:41.177166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:41.177300 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:41.177874 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:41.177890 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:41.179423 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:41.181265 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:41.181270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:41.181281 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:41.181526 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:41.181775 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:41.181778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:41.181779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:41.181904 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:41.182259 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:41.182264 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:41.182542 (+ 278us) client_negotiation.cc:769] Sending connection context
0621 07:51:41.182711 (+ 169us) client_negotiation.cc:240] Negotiation successful
0621 07:51:41.182964 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":122,"threads_started":1}
I20260621 07:51:41.185063 12847 heartbeater.cc:347] Connected to a master server at 127.12.12.62:42509
I20260621 07:51:41.185284 12847 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:41.185767 12847 heartbeater.cc:511] Master 127.12.12.62:42509 requested a full tablet report, sending...
I20260621 07:51:41.186794 12388 ts_manager.cc:194] Registered new tserver with Master: 2eb6a908f86e4071a031b0522be36836 (127.12.12.3:43697)
I20260621 07:51:41.187357 12388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:45929
I20260621 07:51:41.196949 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 07:51:41 dist-test-slave-lbpt krb5kdc[12343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 07:51:41 dist-test-slave-lbpt krb5kdc[12343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.12.1@KRBTEST.COM
I20260621 07:51:41.224645 12854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:41.214100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.12.12.1:36919)
0621 07:51:41.214466 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:41.214485 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:41.214586 (+ 101us) server_negotiation.cc:407] Connection header received
0621 07:51:41.214860 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:41.214868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:41.215084 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:41.215224 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:41.215960 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:41.216762 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:41.217623 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:41.218076 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:41.218174 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:41.220860 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:41.220872 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:41.220885 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:41.220912 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:41.222864 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:41.223343 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:41.223349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:41.223350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:41.223434 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:41.223667 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:41.223669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:41.223670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:41.223957 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:41.224146 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:41.224359 (+ 213us) server_negotiation.cc:299] Negotiation successful
0621 07:51:41.224489 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":107,"threads_started":1}
W20260621 07:51:41.225771 12493 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42994
Jun 21 07:51:41 dist-test-slave-lbpt krb5kdc[12343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 07:51:41 dist-test-slave-lbpt krb5kdc[12343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.1@KRBTEST.COM
I20260621 07:51:41.251659 12854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:41.243425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.12.12.1:36919)
0621 07:51:41.243588 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:41.243593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:41.243726 (+ 133us) server_negotiation.cc:407] Connection header received
0621 07:51:41.243901 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:41.243905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:41.243953 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:41.244038 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:41.244715 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:41.245240 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:41.246057 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:41.246306 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:41.246412 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:41.248869 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:41.248872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:41.248874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:41.248901 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:41.250151 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:41.250581 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:41.250584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:41.250585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:41.250664 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:41.250876 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:41.250878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:41.250879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:41.251034 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:41.251132 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:41.251377 (+ 245us) server_negotiation.cc:299] Negotiation successful
0621 07:51:41.251477 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 07:51:41.252933 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 12450
I20260621 07:51:41.258528 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 12584
I20260621 07:51:41.264233 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 12718
I20260621 07:51:41.269544 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 12356
2026-06-21T07:51:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2198 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:41 dist-test-slave-lbpt krb5kdc[12861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:41 dist-test-slave-lbpt krb5kdc[12861](info): set up 2 sockets
Jun 21 07:51:41 dist-test-slave-lbpt krb5kdc[12861](info): commencing operation
krb5kdc: starting...
W20260621 07:51:42.321998 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:51:42 dist-test-slave-lbpt krb5kdc[12861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:51:42.442523 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:38385
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:38385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:42.556082 12874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:42.556362 12874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:42.556404 12874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:42.560559 12874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:42.560617 12874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:42.560642 12874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:42.560701 12874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:42.560734 12874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:42.566040 12874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:38385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:38385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:42.567215 12874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:42.568266 12874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:42.576960 12880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:42.576960 12879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:42.576961 12882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:42.577036 12874 server_base.cc:1034] running on GCE node
I20260621 07:51:42.577809 12874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:42.579021 12874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:42.580224 12874 hybrid_clock.cc:648] HybridClock initialized: now 1782028302580189 us; error 48 us; skew 500 ppm
Jun 21 07:51:42 dist-test-slave-lbpt krb5kdc[12861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028302, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:42.583266 12874 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:51:42.584264 12874 webserver.cc:460] Webserver started at http://127.12.12.62:33369/ using document root <none> and password file <none>
I20260621 07:51:42.584856 12874 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:42.584946 12874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:42.585191 12874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:42.587333 12874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "c34a2cee91d548f08eeaf4c37a644878"
format_stamp: "Formatted at 2026-06-21 07:51:42 on dist-test-slave-lbpt"
server_key: "90b3fefd80c3f0b2b975389d5df924b8"
server_key_iv: "e0f1415de615b8bbcb5cd97f45993161"
server_key_version: "encryptionkey@0"
I20260621 07:51:42.587898 12874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "c34a2cee91d548f08eeaf4c37a644878"
format_stamp: "Formatted at 2026-06-21 07:51:42 on dist-test-slave-lbpt"
server_key: "90b3fefd80c3f0b2b975389d5df924b8"
server_key_iv: "e0f1415de615b8bbcb5cd97f45993161"
server_key_version: "encryptionkey@0"
I20260621 07:51:42.591612 12874 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260621 07:51:42.594161 12889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:42.595500 12874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:51:42.595643 12874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "c34a2cee91d548f08eeaf4c37a644878"
format_stamp: "Formatted at 2026-06-21 07:51:42 on dist-test-slave-lbpt"
server_key: "90b3fefd80c3f0b2b975389d5df924b8"
server_key_iv: "e0f1415de615b8bbcb5cd97f45993161"
server_key_version: "encryptionkey@0"
I20260621 07:51:42.595759 12874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:42.609095 12874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:42.612649 12874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:42.612856 12874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:42.620291 12874 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:38385
I20260621 07:51:42.620299 12941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:38385 every 8 connection(s)
I20260621 07:51:42.621268 12874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:42.624080 12942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:42.628626 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 12874
I20260621 07:51:42.628744 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:51:42.629024 12336 external_mini_cluster.cc:1428] Setting key ba99d4d7aae9da98935f12b777d30e92
I20260621 07:51:42.629935 12942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878: Bootstrap starting.
I20260621 07:51:42.632261 12942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:42.633206 12942 log.cc:824] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:42.635354 12942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878: No bootstrap required, opened a new log
Jun 21 07:51:42 dist-test-slave-lbpt krb5kdc[12861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:42.640102 12942 raft_consensus.cc:348] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34a2cee91d548f08eeaf4c37a644878" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38385 } }
I20260621 07:51:42.640388 12942 raft_consensus.cc:374] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:42.640444 12942 raft_consensus.cc:729] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34a2cee91d548f08eeaf4c37a644878, State: Initialized, Role: FOLLOWER
I20260621 07:51:42.640924 12942 consensus_queue.cc:260] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34a2cee91d548f08eeaf4c37a644878" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38385 } }
I20260621 07:51:42.641080 12942 raft_consensus.cc:388] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:42.641150 12942 raft_consensus.cc:482] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:42.641299 12942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:42.642514 12942 raft_consensus.cc:504] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34a2cee91d548f08eeaf4c37a644878" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38385 } }
I20260621 07:51:42.642884 12942 leader_election.cc:302] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c34a2cee91d548f08eeaf4c37a644878; no voters:
I20260621 07:51:42.643239 12942 leader_election.cc:288] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:42.643345 12947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:42.643567 12947 raft_consensus.cc:686] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [term 1 LEADER]: Becoming Leader. State: Replica: c34a2cee91d548f08eeaf4c37a644878, State: Running, Role: LEADER
I20260621 07:51:42.643922 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:42.630478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.12.12.62:38385)
0621 07:51:42.630919 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:42.630935 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:42.630969 (+ 34us) server_negotiation.cc:407] Connection header received
0621 07:51:42.631596 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:42.631613 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:42.631941 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:42.632230 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:42.633097 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:42.634148 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:42.635282 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:42.635794 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:42.635921 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:42.638669 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:42.638677 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:42.638694 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:42.638725 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:42.640656 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:42.641123 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:42.641130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:42.641135 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:42.641299 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:42.641604 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:42.641610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:42.641614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:42.642124 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:42.642320 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:42.642564 (+ 244us) server_negotiation.cc:299] Negotiation successful
0621 07:51:42.642884 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":151,"threads_started":1}
I20260621 07:51:42.643895 12947 consensus_queue.cc:237] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34a2cee91d548f08eeaf4c37a644878" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38385 } }
I20260621 07:51:42.644541 12942 sys_catalog.cc:564] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:42.645515 12948 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c34a2cee91d548f08eeaf4c37a644878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34a2cee91d548f08eeaf4c37a644878" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38385 } } }
I20260621 07:51:42.645645 12948 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:42.645978 12949 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c34a2cee91d548f08eeaf4c37a644878. Latest consensus state: current_term: 1 leader_uuid: "c34a2cee91d548f08eeaf4c37a644878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34a2cee91d548f08eeaf4c37a644878" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38385 } } }
I20260621 07:51:42.646071 12949 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:42.646339 12956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:42.649503 12956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:42.655267 12956 catalog_manager.cc:1347] Generated new cluster ID: 047fc916e4d54c438d79012da2d625fb
I20260621 07:51:42.655344 12956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:42.693405 12956 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:42.694478 12956 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:42.701529 12956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c34a2cee91d548f08eeaf4c37a644878: Generated new TSK 0
I20260621 07:51:42.702343 12956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:51:42.733647 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38385
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:42.847779 12968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:42.848055 12968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:42.848100 12968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:42.852321 12968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:42.852391 12968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:42.852454 12968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:51:42.857731 12968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38385
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:42.858832 12968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:42.859927 12968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:42.870018 12976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:42.870024 12973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:42.870033 12974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:42.870376 12968 server_base.cc:1034] running on GCE node
I20260621 07:51:42.870865 12968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:42.871536 12968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:42.872709 12968 hybrid_clock.cc:648] HybridClock initialized: now 1782028302872670 us; error 48 us; skew 500 ppm
Jun 21 07:51:42 dist-test-slave-lbpt krb5kdc[12861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028302, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:42.875514 12968 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:51:42.876401 12968 webserver.cc:460] Webserver started at http://127.12.12.1:38225/ using document root <none> and password file <none>
I20260621 07:51:42.876925 12968 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:42.876982 12968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:42.877153 12968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:42.879210 12968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "ce4c2b5fa19d463f90750550ff8ac7a3"
format_stamp: "Formatted at 2026-06-21 07:51:42 on dist-test-slave-lbpt"
server_key: "84aefacd8c9f61a17e1c1f63e18e6705"
server_key_iv: "8c4cfadac3adbbb7358aaa5bb7814953"
server_key_version: "encryptionkey@0"
I20260621 07:51:42.879804 12968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "ce4c2b5fa19d463f90750550ff8ac7a3"
format_stamp: "Formatted at 2026-06-21 07:51:42 on dist-test-slave-lbpt"
server_key: "84aefacd8c9f61a17e1c1f63e18e6705"
server_key_iv: "8c4cfadac3adbbb7358aaa5bb7814953"
server_key_version: "encryptionkey@0"
I20260621 07:51:42.883448 12968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:51:42.885852 12983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:42.887102 12968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260621 07:51:42.887218 12968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "ce4c2b5fa19d463f90750550ff8ac7a3"
format_stamp: "Formatted at 2026-06-21 07:51:42 on dist-test-slave-lbpt"
server_key: "84aefacd8c9f61a17e1c1f63e18e6705"
server_key_iv: "8c4cfadac3adbbb7358aaa5bb7814953"
server_key_version: "encryptionkey@0"
I20260621 07:51:42.887307 12968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:42.916540 12968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:42.919847 12968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:42.920004 12968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:42.920648 12968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:42.921459 12968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:42.921504 12968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:42.921545 12968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:42.921561 12968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:42.930619 12968 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:45899
I20260621 07:51:42.930639 13096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:45899 every 8 connection(s)
I20260621 07:51:42.931536 12968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:51:42.940820 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 12968
I20260621 07:51:42.940915 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:51:42.941200 12336 external_mini_cluster.cc:1428] Setting key ae84d0e7a6b54b8b54363549cba44d2f
Jun 21 07:51:42 dist-test-slave-lbpt krb5kdc[12861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028302, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:42.946220 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:42.933393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:60311 (local address 127.12.12.62:38385)
0621 07:51:42.933532 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:42.933536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:42.934438 (+ 902us) server_negotiation.cc:407] Connection header received
0621 07:51:42.935793 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:42.935796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:42.935852 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:42.935933 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:42.937145 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:42.937683 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:42.938923 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:42.939180 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:42.939295 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:42.942036 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:42.942039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:42.942041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:42.942065 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:42.943849 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:42.944370 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:42.944375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:42.944380 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:42.944481 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:42.944948 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:42.944950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:42.944952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:42.945125 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:42.945205 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:42.945937 (+ 732us) server_negotiation.cc:299] Negotiation successful
0621 07:51:42.946061 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:51:42.947134 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:42.933823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38385 (local address 127.12.12.1:60311)
0621 07:51:42.934266 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:42.934310 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:42.935570 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:42.936103 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:42.936110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:42.936474 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:42.936978 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:42.936992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:42.937818 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:42.937822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:42.938797 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:42.938807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:42.939339 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:42.939343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:42.939515 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:42.940070 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:42.940086 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:42.941862 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:42.943969 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:42.943974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:42.943987 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:42.944245 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:42.944589 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:42.944594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:42.944597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:42.944809 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:42.945230 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:42.945238 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:42.945656 (+ 418us) client_negotiation.cc:769] Sending connection context
0621 07:51:42.945900 (+ 244us) client_negotiation.cc:240] Negotiation successful
0621 07:51:42.946284 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":117,"threads_started":1}
I20260621 07:51:42.948336 13097 heartbeater.cc:347] Connected to a master server at 127.12.12.62:38385
I20260621 07:51:42.948586 13097 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:42.949066 13097 heartbeater.cc:511] Master 127.12.12.62:38385 requested a full tablet report, sending...
I20260621 07:51:42.950829 12906 ts_manager.cc:194] Registered new tserver with Master: ce4c2b5fa19d463f90750550ff8ac7a3 (127.12.12.1:45899)
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
I20260621 07:51:42.952126 12906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:60311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:51:42.969538 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38385
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:43.081687 13102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:43.081954 13102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:43.081998 13102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:43.086021 13102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:43.086087 13102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:43.086146 13102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:51:43.091679 13102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38385
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:43.092723 13102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:43.093699 13102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:43.103150 13110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:43.103308 13102 server_base.cc:1034] running on GCE node
W20260621 07:51:43.103174 13108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:43.103150 13107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:43.103890 13102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:43.104566 13102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:43.105772 13102 hybrid_clock.cc:648] HybridClock initialized: now 1782028303105756 us; error 28 us; skew 500 ppm
Jun 21 07:51:43 dist-test-slave-lbpt krb5kdc[12861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028303, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:43.108712 13102 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:51:43.109547 13102 webserver.cc:460] Webserver started at http://127.12.12.2:37049/ using document root <none> and password file <none>
I20260621 07:51:43.110116 13102 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:43.110208 13102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:43.110513 13102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:43.112521 13102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "4f3c0a219a9f4778808bee01a06225b3"
format_stamp: "Formatted at 2026-06-21 07:51:43 on dist-test-slave-lbpt"
server_key: "1378f6a5fd3c14e1b2b05eb2f40a3edb"
server_key_iv: "a91e7b688a9afead2afcd490b8861e6c"
server_key_version: "encryptionkey@0"
I20260621 07:51:43.113034 13102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "4f3c0a219a9f4778808bee01a06225b3"
format_stamp: "Formatted at 2026-06-21 07:51:43 on dist-test-slave-lbpt"
server_key: "1378f6a5fd3c14e1b2b05eb2f40a3edb"
server_key_iv: "a91e7b688a9afead2afcd490b8861e6c"
server_key_version: "encryptionkey@0"
I20260621 07:51:43.116878 13102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:51:43.119270 13117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:43.120489 13102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:43.120586 13102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "4f3c0a219a9f4778808bee01a06225b3"
format_stamp: "Formatted at 2026-06-21 07:51:43 on dist-test-slave-lbpt"
server_key: "1378f6a5fd3c14e1b2b05eb2f40a3edb"
server_key_iv: "a91e7b688a9afead2afcd490b8861e6c"
server_key_version: "encryptionkey@0"
I20260621 07:51:43.120674 13102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:43.142911 13102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:43.146343 13102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:43.146627 13102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:43.147307 13102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:43.148080 13102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:43.148125 13102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:43.148164 13102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:43.148180 13102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:43.157279 13102 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:35967
I20260621 07:51:43.157299 13230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:35967 every 8 connection(s)
I20260621 07:51:43.158207 13102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:51:43.166154 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 13102
I20260621 07:51:43.166273 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:51:43.166749 12336 external_mini_cluster.cc:1428] Setting key 3952dc8fd7163ecb989a7498de2014f1
Jun 21 07:51:43 dist-test-slave-lbpt krb5kdc[12861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028303, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:43.172398 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.160114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:38393 (local address 127.12.12.62:38385)
0621 07:51:43.160254 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:43.160258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:43.161078 (+ 820us) server_negotiation.cc:407] Connection header received
0621 07:51:43.162488 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:43.162492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:43.162548 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:43.162625 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:43.163816 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.164340 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.165518 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.165732 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.165814 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.168687 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:43.168690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:43.168691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:43.168713 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:43.170192 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:43.170839 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:43.170844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:43.170846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:43.170930 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:43.171282 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:43.171285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:43.171287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:43.171470 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:43.171577 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:43.172081 (+ 504us) server_negotiation.cc:299] Negotiation successful
0621 07:51:43.172218 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 07:51:43.173202 13233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.160499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38385 (local address 127.12.12.2:38393)
0621 07:51:43.160921 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:43.160963 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:43.162241 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:43.162793 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:43.162800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:43.163164 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:43.163679 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:43.163693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.164490 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.164493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:43.165402 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:43.165412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.165920 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.165926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:43.166156 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.166832 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:43.166854 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:43.168492 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:43.170338 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:43.170344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:43.170359 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:43.170709 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:43.171029 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:43.171032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:43.171034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:43.171178 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:43.171575 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:43.171582 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:43.171841 (+ 259us) client_negotiation.cc:769] Sending connection context
0621 07:51:43.172030 (+ 189us) client_negotiation.cc:240] Negotiation successful
0621 07:51:43.172355 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":112,"threads_started":1}
I20260621 07:51:43.174567 13231 heartbeater.cc:347] Connected to a master server at 127.12.12.62:38385
I20260621 07:51:43.174818 13231 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:43.175321 13231 heartbeater.cc:511] Master 127.12.12.62:38385 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
I20260621 07:51:43.176474 12906 ts_manager.cc:194] Registered new tserver with Master: 4f3c0a219a9f4778808bee01a06225b3 (127.12.12.2:35967)
I20260621 07:51:43.177066 12906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:38393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:51:43.193715 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38385
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:43.307495 13236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:43.307798 13236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:43.307844 13236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:43.312127 13236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:43.312191 13236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:43.312252 13236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:51:43.317550 13236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38385
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:43.318570 13236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:43.319681 13236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:43.329157 13244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:43.329175 13242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:43.329159 13241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:43.329555 13236 server_base.cc:1034] running on GCE node
I20260621 07:51:43.329950 13236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:43.330624 13236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:43.331808 13236 hybrid_clock.cc:648] HybridClock initialized: now 1782028303331794 us; error 27 us; skew 500 ppm
Jun 21 07:51:43 dist-test-slave-lbpt krb5kdc[12861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028303, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:43.334695 13236 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:51:43.335532 13236 webserver.cc:460] Webserver started at http://127.12.12.3:32871/ using document root <none> and password file <none>
I20260621 07:51:43.336081 13236 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:43.336148 13236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:43.336354 13236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:43.338358 13236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "456b7060de3e4b5499b701a1b841f067"
format_stamp: "Formatted at 2026-06-21 07:51:43 on dist-test-slave-lbpt"
server_key: "78c453b2c034302652c9a9888432c8ab"
server_key_iv: "0ab377547caf01e5d8881cafee5be302"
server_key_version: "encryptionkey@0"
I20260621 07:51:43.338972 13236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "456b7060de3e4b5499b701a1b841f067"
format_stamp: "Formatted at 2026-06-21 07:51:43 on dist-test-slave-lbpt"
server_key: "78c453b2c034302652c9a9888432c8ab"
server_key_iv: "0ab377547caf01e5d8881cafee5be302"
server_key_version: "encryptionkey@0"
I20260621 07:51:43.343086 13236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:51:43.345829 13251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:43.347173 13236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:51:43.347312 13236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "456b7060de3e4b5499b701a1b841f067"
format_stamp: "Formatted at 2026-06-21 07:51:43 on dist-test-slave-lbpt"
server_key: "78c453b2c034302652c9a9888432c8ab"
server_key_iv: "0ab377547caf01e5d8881cafee5be302"
server_key_version: "encryptionkey@0"
I20260621 07:51:43.347431 13236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:43.372828 13236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:43.376287 13236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:43.376489 13236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:43.377106 13236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:43.377974 13236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:43.378044 13236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:43.378113 13236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:43.378158 13236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:43.387426 13236 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:33237
I20260621 07:51:43.387445 13364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:33237 every 8 connection(s)
I20260621 07:51:43.388365 13236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:51:43.390686 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 13236
I20260621 07:51:43.390818 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:51:43.391175 12336 external_mini_cluster.cc:1428] Setting key 52ee7998ea1e1a0c78e383a2ae18e281
Jun 21 07:51:43 dist-test-slave-lbpt krb5kdc[12861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028303, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:43.403131 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.390304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:58273 (local address 127.12.12.62:38385)
0621 07:51:43.390454 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:43.390458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:43.391525 (+ 1067us) server_negotiation.cc:407] Connection header received
0621 07:51:43.393151 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:43.393154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:43.393211 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:43.393287 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:43.394636 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.395178 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.396338 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.396577 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.396696 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.399343 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:43.399346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:43.399348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:43.399370 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:43.400887 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:43.401457 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:43.401460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:43.401461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:43.401533 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:43.401918 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:43.401920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:43.401921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:43.402089 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:43.402162 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:43.402863 (+ 701us) server_negotiation.cc:299] Negotiation successful
0621 07:51:43.402980 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 07:51:43.404174 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.390763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38385 (local address 127.12.12.3:58273)
0621 07:51:43.391348 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:43.391408 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:43.392940 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:43.393467 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:43.393473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:43.393882 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:43.394490 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:43.394512 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.395317 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.395321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:43.396211 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:43.396220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.396728 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.396733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:43.396918 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.397496 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:43.397512 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:43.399172 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:43.401005 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:43.401010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:43.401022 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:43.401341 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:43.401640 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:43.401644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:43.401646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:43.401823 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:43.402201 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:43.402207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:43.402564 (+ 357us) client_negotiation.cc:769] Sending connection context
0621 07:51:43.402767 (+ 203us) client_negotiation.cc:240] Negotiation successful
0621 07:51:43.403054 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":150,"threads_started":1}
I20260621 07:51:43.405344 13365 heartbeater.cc:347] Connected to a master server at 127.12.12.62:38385
I20260621 07:51:43.405637 13365 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:43.406147 13365 heartbeater.cc:511] Master 127.12.12.62:38385 requested a full tablet report, sending...
I20260621 07:51:43.407236 12905 ts_manager.cc:194] Registered new tserver with Master: 456b7060de3e4b5499b701a1b841f067 (127.12.12.3:33237)
I20260621 07:51:43.407866 12905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:58273
I20260621 07:51:43.416981 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:51:43.428258 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.420596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38534 (local address 127.12.12.62:38385)
0621 07:51:43.420724 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:43.420728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:43.420910 (+ 182us) server_negotiation.cc:407] Connection header received
0621 07:51:43.421033 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:43.421036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:43.421077 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:43.421146 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:43.421847 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.422364 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.423276 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.423476 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.423584 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.424963 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:43.424966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:43.424968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:43.424994 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:43.426518 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:43.426891 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:43.426893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:43.426895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:43.426963 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:43.427235 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:43.427237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:43.427238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:43.427455 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:43.427554 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:43.428026 (+ 472us) server_negotiation.cc:299] Negotiation successful
0621 07:51:43.428128 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:51:43.433274 12905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:51:43.436465 12905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:51:43.451596 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.446056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.12.12.3:33237)
0621 07:51:43.446519 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:43.446534 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:43.446555 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:51:43.446636 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:43.446640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:43.446868 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:43.447019 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:43.447602 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.448798 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.450222 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.450902 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.451028 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.451231 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:43.451343 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 07:51:43.451433 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":98,"threads_started":1}
I20260621 07:51:43.451722 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.445409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:33237 (local address 127.0.0.1:33558)
0621 07:51:43.446053 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:43.446105 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:43.446290 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:43.447210 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:43.447216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:43.447240 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:43.447465 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:43.447474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.448965 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.448968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:43.450100 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:43.450110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.451026 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.451029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:43.451141 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.451181 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 07:51:43.451316 (+ 135us) client_negotiation.cc:240] Negotiation successful
0621 07:51:43.451433 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":112,"spinlock_wait_cycles":10496,"thread_start_us":89,"threads_started":1}
I20260621 07:51:43.452046 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.446182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39594 (local address 127.12.12.2:35967)
0621 07:51:43.446660 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:43.446674 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:43.447000 (+ 326us) server_negotiation.cc:407] Connection header received
0621 07:51:43.447086 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:43.447090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:43.447251 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:43.447394 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:43.447647 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.448491 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.450107 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.450984 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.451102 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.451275 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:43.451787 (+ 512us) server_negotiation.cc:299] Negotiation successful
0621 07:51:43.451887 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":88,"threads_started":1}
I20260621 07:51:43.452286 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.445636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:35967 (local address 127.0.0.1:39594)
0621 07:51:43.446807 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:43.446820 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:43.446907 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:43.447385 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:43.447390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:43.447408 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:43.447521 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:43.447530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.448615 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.448621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:43.449975 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:43.449985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.451337 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.451340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:43.451465 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.451487 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:51:43.451558 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 07:51:43.451640 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":22,"thread_start_us":96,"threads_started":1}
I20260621 07:51:43.452419 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.445330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.12.12.1:45899)
0621 07:51:43.445856 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:43.445877 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:43.446457 (+ 580us) server_negotiation.cc:407] Connection header received
0621 07:51:43.446548 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:43.446554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:43.446926 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:43.447070 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:43.447622 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.448798 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.450260 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.451060 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.451249 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.451372 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:43.452024 (+ 652us) server_negotiation.cc:299] Negotiation successful
0621 07:51:43.452127 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":140,"threads_started":1}
I20260621 07:51:43.452610 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.445214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:45899 (local address 127.0.0.1:56436)
0621 07:51:43.446054 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:43.446107 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:43.446291 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:43.447241 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:43.447244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:43.447262 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:43.447484 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:43.447492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.448918 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.448922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:43.450146 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:43.450153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.451417 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.451421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:43.451534 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.451563 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 07:51:43.451620 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 07:51:43.451668 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":110,"spinlock_wait_cycles":77952,"thread_start_us":80,"threads_started":1}
I20260621 07:51:43.455068 13031 tablet_service.cc:1467] Processing CreateTablet for tablet 9e927d0e2ff545af8ac220a1c6cf8c46 (DEFAULT_TABLE table=test-table [id=7805f6c4ff3441a3b32cf58ddcc77004]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:43.455088 13299 tablet_service.cc:1467] Processing CreateTablet for tablet 9e927d0e2ff545af8ac220a1c6cf8c46 (DEFAULT_TABLE table=test-table [id=7805f6c4ff3441a3b32cf58ddcc77004]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:43.455265 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 9e927d0e2ff545af8ac220a1c6cf8c46 (DEFAULT_TABLE table=test-table [id=7805f6c4ff3441a3b32cf58ddcc77004]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:43.455964 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e927d0e2ff545af8ac220a1c6cf8c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:43.456086 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e927d0e2ff545af8ac220a1c6cf8c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:43.456086 13299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e927d0e2ff545af8ac220a1c6cf8c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:43.463873 13382 tablet_bootstrap.cc:492] T 9e927d0e2ff545af8ac220a1c6cf8c46 P ce4c2b5fa19d463f90750550ff8ac7a3: Bootstrap starting.
I20260621 07:51:43.464152 13383 tablet_bootstrap.cc:492] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 4f3c0a219a9f4778808bee01a06225b3: Bootstrap starting.
I20260621 07:51:43.464794 13384 tablet_bootstrap.cc:492] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067: Bootstrap starting.
I20260621 07:51:43.466290 13382 tablet_bootstrap.cc:654] T 9e927d0e2ff545af8ac220a1c6cf8c46 P ce4c2b5fa19d463f90750550ff8ac7a3: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:43.466749 13383 tablet_bootstrap.cc:654] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 4f3c0a219a9f4778808bee01a06225b3: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:43.467046 13384 tablet_bootstrap.cc:654] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:43.467270 13382 log.cc:824] T 9e927d0e2ff545af8ac220a1c6cf8c46 P ce4c2b5fa19d463f90750550ff8ac7a3: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:43.467537 13383 log.cc:824] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 4f3c0a219a9f4778808bee01a06225b3: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:43.467837 13384 log.cc:824] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:43.469347 13382 tablet_bootstrap.cc:492] T 9e927d0e2ff545af8ac220a1c6cf8c46 P ce4c2b5fa19d463f90750550ff8ac7a3: No bootstrap required, opened a new log
I20260621 07:51:43.469449 13384 tablet_bootstrap.cc:492] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067: No bootstrap required, opened a new log
I20260621 07:51:43.469523 13383 tablet_bootstrap.cc:492] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 4f3c0a219a9f4778808bee01a06225b3: No bootstrap required, opened a new log
I20260621 07:51:43.469537 13382 ts_tablet_manager.cc:1397] T 9e927d0e2ff545af8ac220a1c6cf8c46 P ce4c2b5fa19d463f90750550ff8ac7a3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:51:43.469575 13384 ts_tablet_manager.cc:1397] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:51:43.469662 13383 ts_tablet_manager.cc:1397] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 4f3c0a219a9f4778808bee01a06225b3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260621 07:51:43.472426 13384 raft_consensus.cc:348] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c2b5fa19d463f90750550ff8ac7a3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45899 } } peers { permanent_uuid: "456b7060de3e4b5499b701a1b841f067" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33237 } } peers { permanent_uuid: "4f3c0a219a9f4778808bee01a06225b3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 35967 } }
I20260621 07:51:43.472687 13384 raft_consensus.cc:374] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:43.472816 13384 raft_consensus.cc:729] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456b7060de3e4b5499b701a1b841f067, State: Initialized, Role: FOLLOWER
I20260621 07:51:43.473306 13384 consensus_queue.cc:260] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c2b5fa19d463f90750550ff8ac7a3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45899 } } peers { permanent_uuid: "456b7060de3e4b5499b701a1b841f067" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33237 } } peers { permanent_uuid: "4f3c0a219a9f4778808bee01a06225b3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 35967 } }
I20260621 07:51:43.473443 13383 raft_consensus.cc:348] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 4f3c0a219a9f4778808bee01a06225b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c2b5fa19d463f90750550ff8ac7a3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45899 } } peers { permanent_uuid: "456b7060de3e4b5499b701a1b841f067" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33237 } } peers { permanent_uuid: "4f3c0a219a9f4778808bee01a06225b3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 35967 } }
I20260621 07:51:43.473714 13383 raft_consensus.cc:374] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 4f3c0a219a9f4778808bee01a06225b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:43.473773 13383 raft_consensus.cc:729] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 4f3c0a219a9f4778808bee01a06225b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f3c0a219a9f4778808bee01a06225b3, State: Initialized, Role: FOLLOWER
I20260621 07:51:43.474076 13365 heartbeater.cc:503] Master 127.12.12.62:38385 was elected leader, sending a full tablet report...
I20260621 07:51:43.474175 13383 consensus_queue.cc:260] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 4f3c0a219a9f4778808bee01a06225b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c2b5fa19d463f90750550ff8ac7a3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45899 } } peers { permanent_uuid: "456b7060de3e4b5499b701a1b841f067" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33237 } } peers { permanent_uuid: "4f3c0a219a9f4778808bee01a06225b3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 35967 } }
I20260621 07:51:43.474325 13384 ts_tablet_manager.cc:1428] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:51:43.474910 13231 heartbeater.cc:503] Master 127.12.12.62:38385 was elected leader, sending a full tablet report...
I20260621 07:51:43.474890 13382 raft_consensus.cc:348] T 9e927d0e2ff545af8ac220a1c6cf8c46 P ce4c2b5fa19d463f90750550ff8ac7a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c2b5fa19d463f90750550ff8ac7a3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45899 } } peers { permanent_uuid: "456b7060de3e4b5499b701a1b841f067" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33237 } } peers { permanent_uuid: "4f3c0a219a9f4778808bee01a06225b3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 35967 } }
I20260621 07:51:43.475128 13383 ts_tablet_manager.cc:1428] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 4f3c0a219a9f4778808bee01a06225b3: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260621 07:51:43.475126 13382 raft_consensus.cc:374] T 9e927d0e2ff545af8ac220a1c6cf8c46 P ce4c2b5fa19d463f90750550ff8ac7a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:43.475186 13382 raft_consensus.cc:729] T 9e927d0e2ff545af8ac220a1c6cf8c46 P ce4c2b5fa19d463f90750550ff8ac7a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4c2b5fa19d463f90750550ff8ac7a3, State: Initialized, Role: FOLLOWER
I20260621 07:51:43.475582 13382 consensus_queue.cc:260] T 9e927d0e2ff545af8ac220a1c6cf8c46 P ce4c2b5fa19d463f90750550ff8ac7a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c2b5fa19d463f90750550ff8ac7a3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45899 } } peers { permanent_uuid: "456b7060de3e4b5499b701a1b841f067" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33237 } } peers { permanent_uuid: "4f3c0a219a9f4778808bee01a06225b3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 35967 } }
I20260621 07:51:43.476362 13097 heartbeater.cc:503] Master 127.12.12.62:38385 was elected leader, sending a full tablet report...
I20260621 07:51:43.477289 13382 ts_tablet_manager.cc:1428] T 9e927d0e2ff545af8ac220a1c6cf8c46 P ce4c2b5fa19d463f90750550ff8ac7a3: Time spent starting tablet: real 0.008s user 0.000s sys 0.006s
I20260621 07:51:43.573904 13388 raft_consensus.cc:482] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:51:43.574137 13388 raft_consensus.cc:504] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c2b5fa19d463f90750550ff8ac7a3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45899 } } peers { permanent_uuid: "456b7060de3e4b5499b701a1b841f067" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33237 } } peers { permanent_uuid: "4f3c0a219a9f4778808bee01a06225b3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 35967 } }
I20260621 07:51:43.575502 13388 leader_election.cc:288] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce4c2b5fa19d463f90750550ff8ac7a3 (127.12.12.1:45899), 4f3c0a219a9f4778808bee01a06225b3 (127.12.12.2:35967)
I20260621 07:51:43.580178 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.576009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:44899 (local address 127.12.12.2:35967)
0621 07:51:43.576120 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:43.576127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:43.576371 (+ 244us) server_negotiation.cc:407] Connection header received
0621 07:51:43.576466 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:43.576468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:43.576539 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:43.576610 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:43.576873 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.577538 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.578954 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.579532 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.579642 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.579705 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:43.579980 (+ 275us) server_negotiation.cc:299] Negotiation successful
0621 07:51:43.580049 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:51:43.580178 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.575835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:42827 (local address 127.12.12.1:45899)
0621 07:51:43.575969 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:43.575974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:43.575990 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:51:43.576045 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:43.576050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:43.576102 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:43.576196 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:43.576553 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.577339 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.578860 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.579532 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.579642 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.579705 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:43.579980 (+ 275us) server_negotiation.cc:299] Negotiation successful
0621 07:51:43.580054 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 07:51:43.580271 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.575704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:45899 (local address 127.12.12.3:42827)
0621 07:51:43.575824 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:43.575840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:43.575938 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:43.576257 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:43.576260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:43.576284 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:43.576424 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:43.576433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.577448 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.577450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:43.578737 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:43.578747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.579684 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.579688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:43.579824 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.579880 (+ 56us) client_negotiation.cc:769] Sending connection context
0621 07:51:43.579948 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 07:51:43.580103 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":52}
I20260621 07:51:43.580495 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.575863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:35967 (local address 127.12.12.3:44899)
0621 07:51:43.576264 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:43.576278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:43.576375 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:43.576662 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:43.576664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:43.576684 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:43.576768 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:43.576772 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.577682 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.577685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:43.578845 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:43.578854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.579697 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.579700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:43.579828 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.579869 (+ 41us) client_negotiation.cc:769] Sending connection context
0621 07:51:43.579948 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 07:51:43.580103 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":76,"thread_start_us":123,"threads_started":1}
I20260621 07:51:43.580935 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e927d0e2ff545af8ac220a1c6cf8c46" candidate_uuid: "456b7060de3e4b5499b701a1b841f067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4c2b5fa19d463f90750550ff8ac7a3" is_pre_election: true
I20260621 07:51:43.581270 13051 raft_consensus.cc:2454] T 9e927d0e2ff545af8ac220a1c6cf8c46 P ce4c2b5fa19d463f90750550ff8ac7a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 456b7060de3e4b5499b701a1b841f067 in term 0.
I20260621 07:51:43.581670 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e927d0e2ff545af8ac220a1c6cf8c46" candidate_uuid: "456b7060de3e4b5499b701a1b841f067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3c0a219a9f4778808bee01a06225b3" is_pre_election: true
I20260621 07:51:43.581826 13254 leader_election.cc:302] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 456b7060de3e4b5499b701a1b841f067, ce4c2b5fa19d463f90750550ff8ac7a3; no voters:
I20260621 07:51:43.582044 13185 raft_consensus.cc:2454] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 4f3c0a219a9f4778808bee01a06225b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 456b7060de3e4b5499b701a1b841f067 in term 0.
I20260621 07:51:43.582135 13388 raft_consensus.cc:2781] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:51:43.582223 13388 raft_consensus.cc:482] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:51:43.582281 13388 raft_consensus.cc:3037] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:43.583348 13388 raft_consensus.cc:504] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c2b5fa19d463f90750550ff8ac7a3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45899 } } peers { permanent_uuid: "456b7060de3e4b5499b701a1b841f067" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33237 } } peers { permanent_uuid: "4f3c0a219a9f4778808bee01a06225b3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 35967 } }
I20260621 07:51:43.583727 13388 leader_election.cc:288] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [CANDIDATE]: Term 1 election: Requested vote from peers ce4c2b5fa19d463f90750550ff8ac7a3 (127.12.12.1:45899), 4f3c0a219a9f4778808bee01a06225b3 (127.12.12.2:35967)
I20260621 07:51:43.584049 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e927d0e2ff545af8ac220a1c6cf8c46" candidate_uuid: "456b7060de3e4b5499b701a1b841f067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4c2b5fa19d463f90750550ff8ac7a3"
I20260621 07:51:43.584123 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e927d0e2ff545af8ac220a1c6cf8c46" candidate_uuid: "456b7060de3e4b5499b701a1b841f067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3c0a219a9f4778808bee01a06225b3"
I20260621 07:51:43.584184 13051 raft_consensus.cc:3037] T 9e927d0e2ff545af8ac220a1c6cf8c46 P ce4c2b5fa19d463f90750550ff8ac7a3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:43.584298 13185 raft_consensus.cc:3037] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 4f3c0a219a9f4778808bee01a06225b3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:43.585259 13051 raft_consensus.cc:2454] T 9e927d0e2ff545af8ac220a1c6cf8c46 P ce4c2b5fa19d463f90750550ff8ac7a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 456b7060de3e4b5499b701a1b841f067 in term 1.
I20260621 07:51:43.585398 13185 raft_consensus.cc:2454] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 4f3c0a219a9f4778808bee01a06225b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 456b7060de3e4b5499b701a1b841f067 in term 1.
I20260621 07:51:43.585640 13254 leader_election.cc:302] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 456b7060de3e4b5499b701a1b841f067, ce4c2b5fa19d463f90750550ff8ac7a3; no voters:
I20260621 07:51:43.585847 13388 raft_consensus.cc:2781] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:43.586017 13388 raft_consensus.cc:686] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [term 1 LEADER]: Becoming Leader. State: Replica: 456b7060de3e4b5499b701a1b841f067, State: Running, Role: LEADER
I20260621 07:51:43.586346 13388 consensus_queue.cc:237] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c2b5fa19d463f90750550ff8ac7a3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45899 } } peers { permanent_uuid: "456b7060de3e4b5499b701a1b841f067" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33237 } } peers { permanent_uuid: "4f3c0a219a9f4778808bee01a06225b3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 35967 } }
I20260621 07:51:43.589491 12905 catalog_manager.cc:5495] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067 reported cstate change: term changed from 0 to 1, leader changed from <none> to 456b7060de3e4b5499b701a1b841f067 (127.12.12.3). New cstate: current_term: 1 leader_uuid: "456b7060de3e4b5499b701a1b841f067" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c2b5fa19d463f90750550ff8ac7a3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "456b7060de3e4b5499b701a1b841f067" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f3c0a219a9f4778808bee01a06225b3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 35967 } health_report { overall_health: UNKNOWN } } }
Jun 21 07:51:43 dist-test-slave-lbpt krb5kdc[12861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.1@KRBTEST.COM
I20260621 07:51:43.605755 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.596879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56442 (local address 127.12.12.1:45899)
0621 07:51:43.597055 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:43.597064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:43.597140 (+ 76us) server_negotiation.cc:407] Connection header received
0621 07:51:43.597248 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:43.597250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:43.597291 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:43.597360 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:43.597988 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.598562 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.599394 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.599611 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.599692 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.602249 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:43.602261 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:43.602276 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:43.602303 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:43.604104 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:43.604519 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:43.604525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:43.604526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:43.604616 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:43.604843 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:43.604845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:43.604846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:43.605087 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:43.605245 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:43.605406 (+ 161us) server_negotiation.cc:299] Negotiation successful
0621 07:51:43.605502 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Jun 21 07:51:43 dist-test-slave-lbpt krb5kdc[12861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 07:51:43 dist-test-slave-lbpt krb5kdc[12861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.12.1@KRBTEST.COM
I20260621 07:51:43.633205 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.623948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.12.12.1:45899)
0621 07:51:43.624105 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:43.624110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:43.624327 (+ 217us) server_negotiation.cc:407] Connection header received
0621 07:51:43.624504 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:43.624508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:43.624551 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:43.624638 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:43.625397 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.626220 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.627139 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.627508 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.627606 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.630092 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:43.630097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:43.630099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:43.630127 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:43.631679 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:43.632053 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:43.632057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:43.632058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:43.632142 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:43.632389 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:43.632392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:43.632393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:43.632577 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:43.632695 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:43.632929 (+ 234us) server_negotiation.cc:299] Negotiation successful
0621 07:51:43.633046 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260621 07:51:43.639350 13366 tablet.cc:2369] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 456b7060de3e4b5499b701a1b841f067: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 07:51:43 dist-test-slave-lbpt krb5kdc[12861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 07:51:43 dist-test-slave-lbpt krb5kdc[12861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.1@KRBTEST.COM
I20260621 07:51:43.658304 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:43.650301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.12.12.1:45899)
0621 07:51:43.650501 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:43.650505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:43.650574 (+ 69us) server_negotiation.cc:407] Connection header received
0621 07:51:43.650699 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:43.650702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:43.650751 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:43.650822 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:43.651562 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.652131 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.652930 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:43.653135 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:43.653212 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:43.655598 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:43.655601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:43.655605 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:43.655627 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:43.656871 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:43.657253 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:43.657255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:43.657256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:43.657327 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:43.657540 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:43.657542 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:43.657543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:43.657694 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:43.657789 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:43.657995 (+ 206us) server_negotiation.cc:299] Negotiation successful
0621 07:51:43.658087 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260621 07:51:43.659361 13232 tablet.cc:2369] T 9e927d0e2ff545af8ac220a1c6cf8c46 P 4f3c0a219a9f4778808bee01a06225b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:51:43.659788 13011 tablet_service.cc:2908] Rejecting scan request for tablet 9e927d0e2ff545af8ac220a1c6cf8c46: Uninitialized: clean time has not yet been initialized
I20260621 07:51:43.661827 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 12968
I20260621 07:51:43.667271 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 13102
I20260621 07:51:43.672616 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 13236
I20260621 07:51:43.677969 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 12874
2026-06-21T07:51:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2408 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:43 dist-test-slave-lbpt krb5kdc[13406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:43 dist-test-slave-lbpt krb5kdc[13406](info): set up 2 sockets
Jun 21 07:51:43 dist-test-slave-lbpt krb5kdc[13406](info): commencing operation
krb5kdc: starting...
W20260621 07:51:44.722082 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:51:44 dist-test-slave-lbpt krb5kdc[13406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:51:44.839471 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:34251
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:34251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:44.950034 13419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:44.950349 13419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:44.950436 13419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:44.954432 13419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:44.954526 13419 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 07:51:44.954561 13419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:44.954617 13419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:44.954674 13419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:44.954715 13419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:44.960017 13419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:34251
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:34251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:44.961081 13419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:44.962113 13419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:44.971154 13427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:44.971199 13425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:44.971154 13424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:44.971228 13419 server_base.cc:1034] running on GCE node
I20260621 07:51:44.972024 13419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:44.973187 13419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:44.974365 13419 hybrid_clock.cc:648] HybridClock initialized: now 1782028304974314 us; error 65 us; skew 500 ppm
Jun 21 07:51:44 dist-test-slave-lbpt krb5kdc[13406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028304, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:44.982030 13419 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:51:44.983108 13419 webserver.cc:460] Webserver started at http://127.12.12.62:36155/ using document root <none> and password file <none>
I20260621 07:51:44.983687 13419 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:44.983757 13419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:44.983984 13419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:44.986078 13419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "b8332ae99cfd4cd9a1d34e09eb12705e"
format_stamp: "Formatted at 2026-06-21 07:51:44 on dist-test-slave-lbpt"
server_key: "b1349221ba33f5c6648e9d3fea093e83"
server_key_iv: "6a44a3d310e2611f86a09ebc1dfe05a9"
server_key_version: "encryptionkey@0"
I20260621 07:51:44.986680 13419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "b8332ae99cfd4cd9a1d34e09eb12705e"
format_stamp: "Formatted at 2026-06-21 07:51:44 on dist-test-slave-lbpt"
server_key: "b1349221ba33f5c6648e9d3fea093e83"
server_key_iv: "6a44a3d310e2611f86a09ebc1dfe05a9"
server_key_version: "encryptionkey@0"
I20260621 07:51:44.990485 13419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 07:51:44.993489 13434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:44.995422 13419 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:51:44.995577 13419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "b8332ae99cfd4cd9a1d34e09eb12705e"
format_stamp: "Formatted at 2026-06-21 07:51:44 on dist-test-slave-lbpt"
server_key: "b1349221ba33f5c6648e9d3fea093e83"
server_key_iv: "6a44a3d310e2611f86a09ebc1dfe05a9"
server_key_version: "encryptionkey@0"
I20260621 07:51:44.995730 13419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:45.028877 13419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:45.032569 13419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:45.032785 13419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:45.040907 13419 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:34251
I20260621 07:51:45.040925 13496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:34251 every 8 connection(s)
I20260621 07:51:45.041906 13419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:45.044637 13497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:45.046160 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 13419
I20260621 07:51:45.046253 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:51:45.046571 12336 external_mini_cluster.cc:1428] Setting key 9b1eb80b9019dfec4ea4b715c02314a9
I20260621 07:51:45.050855 13497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e: Bootstrap starting.
I20260621 07:51:45.053442 13497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:45.054472 13497 log.cc:824] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:51:45 dist-test-slave-lbpt krb5kdc[13406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:45.056653 13497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e: No bootstrap required, opened a new log
I20260621 07:51:45.060940 13497 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8332ae99cfd4cd9a1d34e09eb12705e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34251 } }
I20260621 07:51:45.061213 13497 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:45.061277 13497 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8332ae99cfd4cd9a1d34e09eb12705e, State: Initialized, Role: FOLLOWER
I20260621 07:51:45.061771 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.047891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.12.12.62:34251)
0621 07:51:45.048289 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:45.048304 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:45.048339 (+ 35us) server_negotiation.cc:407] Connection header received
0621 07:51:45.048945 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:45.048961 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:45.049284 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:45.049552 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:45.050469 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.051477 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.052626 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.053083 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.053198 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.056024 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:45.056030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:45.056046 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:45.056076 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:45.058767 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:45.059190 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:45.059196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:45.059202 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:45.059344 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:45.059588 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:45.059592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:45.059593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:45.060091 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:45.060323 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:45.060508 (+ 185us) server_negotiation.cc:299] Negotiation successful
0621 07:51:45.060801 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":126,"threads_started":1}
I20260621 07:51:45.061753 13497 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8332ae99cfd4cd9a1d34e09eb12705e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34251 } }
I20260621 07:51:45.061920 13497 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:45.061982 13497 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:45.062080 13497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:45.063189 13497 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8332ae99cfd4cd9a1d34e09eb12705e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34251 } }
I20260621 07:51:45.063537 13497 leader_election.cc:302] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8332ae99cfd4cd9a1d34e09eb12705e; no voters:
I20260621 07:51:45.063898 13497 leader_election.cc:288] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:45.064010 13502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:45.064280 13502 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [term 1 LEADER]: Becoming Leader. State: Replica: b8332ae99cfd4cd9a1d34e09eb12705e, State: Running, Role: LEADER
I20260621 07:51:45.064599 13502 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8332ae99cfd4cd9a1d34e09eb12705e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34251 } }
I20260621 07:51:45.065109 13497 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:45.065704 13502 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8332ae99cfd4cd9a1d34e09eb12705e. Latest consensus state: current_term: 1 leader_uuid: "b8332ae99cfd4cd9a1d34e09eb12705e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8332ae99cfd4cd9a1d34e09eb12705e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34251 } } }
I20260621 07:51:45.065825 13502 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:45.066543 13504 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8332ae99cfd4cd9a1d34e09eb12705e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8332ae99cfd4cd9a1d34e09eb12705e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34251 } } }
I20260621 07:51:45.066650 13504 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:45.066968 13508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:45.070098 13508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:45.075697 13508 catalog_manager.cc:1347] Generated new cluster ID: 02d030ed70ae4d4bb43fe5ae9eed86b2
I20260621 07:51:45.075783 13508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:45.090291 13508 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:45.091626 13508 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:45.101325 13508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e: Generated new TSK 0
I20260621 07:51:45.102165 13508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:51:45.131451 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:34251
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:45.247390 13523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:45.247655 13523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:45.247701 13523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:45.251633 13523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:45.251693 13523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:45.251734 13523 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:51:45.251766 13523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:51:45.256886 13523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.62:34251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:45.257891 13523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:45.258867 13523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:45.268894 13531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:45.268896 13528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:45.268905 13529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:45.269131 13523 server_base.cc:1034] running on GCE node
I20260621 07:51:45.269677 13523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:45.270401 13523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:45.271615 13523 hybrid_clock.cc:648] HybridClock initialized: now 1782028305271597 us; error 34 us; skew 500 ppm
Jun 21 07:51:45 dist-test-slave-lbpt krb5kdc[13406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028305, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:45.274523 13523 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:51:45.275506 13523 webserver.cc:460] Webserver started at http://127.12.12.1:36757/ using document root <none> and password file <none>
I20260621 07:51:45.276098 13523 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:45.276189 13523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:45.276405 13523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:45.278530 13523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "eeb16501b843492eadf65978e1008de8"
format_stamp: "Formatted at 2026-06-21 07:51:45 on dist-test-slave-lbpt"
server_key: "71b4292829ef88e5751813abce7d3680"
server_key_iv: "72464a390512bf1bed4664796a7c8f6c"
server_key_version: "encryptionkey@0"
I20260621 07:51:45.279095 13523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "eeb16501b843492eadf65978e1008de8"
format_stamp: "Formatted at 2026-06-21 07:51:45 on dist-test-slave-lbpt"
server_key: "71b4292829ef88e5751813abce7d3680"
server_key_iv: "72464a390512bf1bed4664796a7c8f6c"
server_key_version: "encryptionkey@0"
I20260621 07:51:45.282869 13523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:51:45.285477 13538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:45.286804 13523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:51:45.286937 13523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "eeb16501b843492eadf65978e1008de8"
format_stamp: "Formatted at 2026-06-21 07:51:45 on dist-test-slave-lbpt"
server_key: "71b4292829ef88e5751813abce7d3680"
server_key_iv: "72464a390512bf1bed4664796a7c8f6c"
server_key_version: "encryptionkey@0"
I20260621 07:51:45.287058 13523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:45.301069 13523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:45.304507 13523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:45.304720 13523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:45.306241 13523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:45.306314 13523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:45.306389 13523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:45.306469 13523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 21 07:51:45 dist-test-slave-lbpt krb5kdc[13406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028305, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:45.317052 13523 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:36481
I20260621 07:51:45.317116 13653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:36481 every 8 connection(s)
I20260621 07:51:45.317938 13523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:51:45.318462 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 13523
I20260621 07:51:45.318572 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:51:45.318876 12336 external_mini_cluster.cc:1428] Setting key 5b9e030203c5a2cf5f323981e4571caa
I20260621 07:51:45.320820 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.307066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:48929 (local address 127.12.12.62:34251)
0621 07:51:45.307226 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:45.307233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:45.308587 (+ 1354us) server_negotiation.cc:407] Connection header received
0621 07:51:45.310071 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:45.310074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:45.310155 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:45.310266 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:45.311524 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.312073 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.313322 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.313554 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.313680 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.316925 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:45.316929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:45.316931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:45.316956 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:45.318689 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:45.319298 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:45.319302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:45.319304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:45.319393 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:45.319733 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:45.319736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:45.319738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:45.319929 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:45.320033 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:45.320520 (+ 487us) server_negotiation.cc:299] Negotiation successful
0621 07:51:45.320644 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 07:51:45.321652 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.307892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:34251 (local address 127.12.12.1:48929)
0621 07:51:45.308418 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:45.308476 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:45.309839 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:45.310490 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:45.310499 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:45.310882 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:45.311368 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.311381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.312211 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.312214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.313200 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.313209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.313721 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.313725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.313903 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.315136 (+ 1233us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:45.315173 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:45.316694 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:45.318818 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:45.318823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:45.318838 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:45.319192 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:45.319493 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:45.319496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:45.319498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:45.319632 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:45.320043 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:45.320051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:45.320287 (+ 236us) client_negotiation.cc:769] Sending connection context
0621 07:51:45.320467 (+ 180us) client_negotiation.cc:240] Negotiation successful
0621 07:51:45.320788 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":153,"threads_started":1}
I20260621 07:51:45.323042 13654 heartbeater.cc:347] Connected to a master server at 127.12.12.62:34251
I20260621 07:51:45.323230 13654 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:45.323836 13654 heartbeater.cc:511] Master 127.12.12.62:34251 requested a full tablet report, sending...
I20260621 07:51:45.325824 13450 ts_manager.cc:194] Registered new tserver with Master: eeb16501b843492eadf65978e1008de8 (127.12.12.1:36481)
I20260621 07:51:45.327354 13450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:48929
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
I20260621 07:51:45.333518 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.325091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:37125 (local address 127.12.12.62:34251)
0621 07:51:45.326533 (+ 1442us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:45.326537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:45.326551 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:51:45.326586 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:45.326589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:45.326629 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:45.326706 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:45.327266 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.327976 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.328817 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.329152 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.329263 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.330341 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:45.330344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:45.330346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:45.330368 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:45.331798 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:45.332280 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:45.332284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:45.332286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:45.332369 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:45.332745 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:45.332748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:45.332749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:45.332920 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:45.333003 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:45.333305 (+ 302us) server_negotiation.cc:299] Negotiation successful
0621 07:51:45.333402 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1366}
I20260621 07:51:45.333648 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.324963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:34251 (local address 127.12.12.1:37125)
0621 07:51:45.325292 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:45.325309 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:45.325408 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:45.326807 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:45.326810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:45.327018 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:45.327127 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.327133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.328146 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.328149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.328702 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.328708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.329268 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.329271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.329368 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.329900 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:45.329912 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:45.330214 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:45.331964 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:45.331969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:45.331974 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:45.332179 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:45.332469 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:45.332471 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:45.332473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:45.332590 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:45.333014 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:45.333020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:45.333130 (+ 110us) client_negotiation.cc:769] Sending connection context
0621 07:51:45.333246 (+ 116us) client_negotiation.cc:240] Negotiation successful
0621 07:51:45.333407 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:51:45.346807 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:34251
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:45.457649 13663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:45.457919 13663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:45.457963 13663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:45.462009 13663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:45.462060 13663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:45.462097 13663 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:51:45.462160 13663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:51:45.467720 13663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.62:34251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:45.468744 13663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:45.469729 13663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:45.480412 13671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:45.480412 13669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:45.480418 13668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:45.480700 13663 server_base.cc:1034] running on GCE node
I20260621 07:51:45.481153 13663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:45.481793 13663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:45.482949 13663 hybrid_clock.cc:648] HybridClock initialized: now 1782028305482928 us; error 33 us; skew 500 ppm
Jun 21 07:51:45 dist-test-slave-lbpt krb5kdc[13406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028305, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:45.485677 13663 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:51:45.486646 13663 webserver.cc:460] Webserver started at http://127.12.12.2:38083/ using document root <none> and password file <none>
I20260621 07:51:45.487203 13663 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:45.487296 13663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:45.487524 13663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:45.489539 13663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "7ace1cfa354946b9b9e5196ac0de1136"
format_stamp: "Formatted at 2026-06-21 07:51:45 on dist-test-slave-lbpt"
server_key: "1b1c93f4617f206d672acbc84c6db812"
server_key_iv: "ec1a69d4a316fdcaa0391f34c80a572e"
server_key_version: "encryptionkey@0"
I20260621 07:51:45.490142 13663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "7ace1cfa354946b9b9e5196ac0de1136"
format_stamp: "Formatted at 2026-06-21 07:51:45 on dist-test-slave-lbpt"
server_key: "1b1c93f4617f206d672acbc84c6db812"
server_key_iv: "ec1a69d4a316fdcaa0391f34c80a572e"
server_key_version: "encryptionkey@0"
I20260621 07:51:45.494221 13663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:51:45.496762 13678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:45.498025 13663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:45.498162 13663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "7ace1cfa354946b9b9e5196ac0de1136"
format_stamp: "Formatted at 2026-06-21 07:51:45 on dist-test-slave-lbpt"
server_key: "1b1c93f4617f206d672acbc84c6db812"
server_key_iv: "ec1a69d4a316fdcaa0391f34c80a572e"
server_key_version: "encryptionkey@0"
I20260621 07:51:45.498278 13663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:45.517884 13663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:45.521401 13663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:45.521616 13663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:45.523360 13663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:45.523434 13663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:45.523509 13663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:45.523550 13663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:51:45 dist-test-slave-lbpt krb5kdc[13406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028305, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:45.534715 13663 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:40541
I20260621 07:51:45.534777 13793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:40541 every 8 connection(s)
I20260621 07:51:45.535661 13663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:51:45.537756 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.524124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:38649 (local address 127.12.12.62:34251)
0621 07:51:45.524244 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:45.524248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:45.525367 (+ 1119us) server_negotiation.cc:407] Connection header received
0621 07:51:45.526379 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:45.526385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:45.526488 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:45.526558 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:45.528062 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.528648 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.529811 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.530067 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.530188 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.533899 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:45.533903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:45.533906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:45.533936 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:45.535711 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:45.536348 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:45.536351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:45.536353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:45.536445 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:45.536805 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:45.536807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:45.536808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:45.536996 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:45.537075 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:45.537525 (+ 450us) server_negotiation.cc:299] Negotiation successful
0621 07:51:45.537638 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 07:51:45.538611 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.524690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:34251 (local address 127.12.12.2:38649)
0621 07:51:45.525196 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:45.525240 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:45.526120 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:45.526976 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:45.526982 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:45.527360 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:45.527905 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.527920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.528798 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.528801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.529673 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.529682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.530245 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.530249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.530456 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.531143 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:45.531167 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:45.533712 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:45.535837 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:45.535843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:45.535858 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:45.536220 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:45.536550 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:45.536553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:45.536555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:45.536714 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:45.537085 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:45.537089 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:45.537317 (+ 228us) client_negotiation.cc:769] Sending connection context
0621 07:51:45.537467 (+ 150us) client_negotiation.cc:240] Negotiation successful
0621 07:51:45.537708 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":143,"threads_started":1}
I20260621 07:51:45.539947 13794 heartbeater.cc:347] Connected to a master server at 127.12.12.62:34251
I20260621 07:51:45.540087 13794 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:45.540656 13794 heartbeater.cc:511] Master 127.12.12.62:34251 requested a full tablet report, sending...
I20260621 07:51:45.541756 13450 ts_manager.cc:194] Registered new tserver with Master: 7ace1cfa354946b9b9e5196ac0de1136 (127.12.12.2:40541)
I20260621 07:51:45.542546 13450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:38649
I20260621 07:51:45.543380 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 13663
I20260621 07:51:45.543490 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:51:45.543774 12336 external_mini_cluster.cc:1428] Setting key 3136b9de4b550a474d00e1e266479238
I20260621 07:51:45.549377 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.542098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:34251 (local address 127.12.12.2:60603)
0621 07:51:45.542399 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:45.542439 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:45.542534 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:45.542817 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:45.542820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:45.543031 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:45.543165 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.543172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.543907 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.543910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.544667 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.544676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.545237 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.545240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.545311 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.545830 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:45.545841 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:45.546134 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:45.547879 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:45.547883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:45.547885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:45.548123 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:45.548426 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:45.548428 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:45.548429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:45.548500 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:45.548986 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:45.548987 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:45.549060 (+ 73us) client_negotiation.cc:769] Sending connection context
0621 07:51:45.549137 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 07:51:45.549240 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":126,"threads_started":1}
I20260621 07:51:45.549710 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.542173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:60603 (local address 127.12.12.62:34251)
0621 07:51:45.542301 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:45.542306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:45.542496 (+ 190us) server_negotiation.cc:407] Connection header received
0621 07:51:45.542634 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:45.542640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:45.542696 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:45.542772 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:45.543282 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.543783 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.544811 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.545122 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.545238 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.546271 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:45.546278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:45.546284 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:45.546316 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:45.547772 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:45.548233 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:45.548239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:45.548244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:45.548335 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:45.548584 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:45.548590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:45.548594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:45.548893 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:45.548998 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:45.549234 (+ 236us) server_negotiation.cc:299] Negotiation successful
0621 07:51:45.549366 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:51:45.570247 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:34251
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:45.682108 13803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:45.682374 13803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:45.682492 13803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:45.686514 13803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:45.686581 13803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:45.686621 13803 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:51:45.686662 13803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:51:45.692147 13803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.62:34251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:45.693171 13803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:45.694152 13803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:45.704614 13811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:45.704634 13808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:45.704790 13803 server_base.cc:1034] running on GCE node
W20260621 07:51:45.704623 13809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:45.705363 13803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:45.706008 13803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:45.707177 13803 hybrid_clock.cc:648] HybridClock initialized: now 1782028305707158 us; error 31 us; skew 500 ppm
Jun 21 07:51:45 dist-test-slave-lbpt krb5kdc[13406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028305, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:45.710134 13803 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:51:45.711128 13803 webserver.cc:460] Webserver started at http://127.12.12.3:41023/ using document root <none> and password file <none>
I20260621 07:51:45.711704 13803 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:45.711795 13803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:45.712026 13803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:45.714069 13803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "17bdfbef628a4fc3819023753ebe8c35"
format_stamp: "Formatted at 2026-06-21 07:51:45 on dist-test-slave-lbpt"
server_key: "2661ccdc239de3401248b31599ec9191"
server_key_iv: "669db6358c729bb9289ce4eaab069a87"
server_key_version: "encryptionkey@0"
I20260621 07:51:45.714651 13803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "17bdfbef628a4fc3819023753ebe8c35"
format_stamp: "Formatted at 2026-06-21 07:51:45 on dist-test-slave-lbpt"
server_key: "2661ccdc239de3401248b31599ec9191"
server_key_iv: "669db6358c729bb9289ce4eaab069a87"
server_key_version: "encryptionkey@0"
I20260621 07:51:45.718281 13803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:51:45.720714 13818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:45.721890 13803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:45.722024 13803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "17bdfbef628a4fc3819023753ebe8c35"
format_stamp: "Formatted at 2026-06-21 07:51:45 on dist-test-slave-lbpt"
server_key: "2661ccdc239de3401248b31599ec9191"
server_key_iv: "669db6358c729bb9289ce4eaab069a87"
server_key_version: "encryptionkey@0"
I20260621 07:51:45.722136 13803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:45.746479 13803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:45.749658 13803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:45.749869 13803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:45.751468 13803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:45.751545 13803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:45.751627 13803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:45.751675 13803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:51:45 dist-test-slave-lbpt krb5kdc[13406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028305, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:45.762972 13803 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:34171
I20260621 07:51:45.763031 13933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:34171 every 8 connection(s)
I20260621 07:51:45.764022 13803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:51:45.765688 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.752452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:51431 (local address 127.12.12.62:34251)
0621 07:51:45.752578 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:45.752581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:45.753727 (+ 1146us) server_negotiation.cc:407] Connection header received
0621 07:51:45.754937 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:45.754941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:45.755001 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:45.755107 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:45.756440 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.757038 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.758267 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.758603 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.758722 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.761895 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:45.761899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:45.761903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:45.761932 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:45.763582 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:45.764151 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:45.764157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:45.764160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:45.764259 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:45.764683 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:45.764689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:45.764693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:45.764933 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:45.765038 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:45.765465 (+ 427us) server_negotiation.cc:299] Negotiation successful
0621 07:51:45.765565 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:51:45.766445 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 13803
I20260621 07:51:45.766556 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:51:45.766733 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.753048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:34251 (local address 127.12.12.3:51431)
0621 07:51:45.753541 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:45.753584 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:45.754714 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:45.755282 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:45.755289 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:45.755628 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:45.756284 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.756316 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.757166 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.757170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.758141 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.758149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.758775 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.758779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.758941 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.759609 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:45.759637 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:45.761678 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:45.763692 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:45.763698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:45.763712 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:45.764038 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:45.764353 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:45.764356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:45.764358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:45.764528 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:45.765038 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:45.765043 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:45.765261 (+ 218us) client_negotiation.cc:769] Sending connection context
0621 07:51:45.765414 (+ 153us) client_negotiation.cc:240] Negotiation successful
0621 07:51:45.765651 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":141,"threads_started":1}
I20260621 07:51:45.766845 12336 external_mini_cluster.cc:1428] Setting key 0c4be6f609b7c96a3862993fb3c6bbbb
I20260621 07:51:45.768146 13934 heartbeater.cc:347] Connected to a master server at 127.12.12.62:34251
I20260621 07:51:45.768275 13934 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:45.768846 13934 heartbeater.cc:511] Master 127.12.12.62:34251 requested a full tablet report, sending...
I20260621 07:51:45.769922 13449 ts_manager.cc:194] Registered new tserver with Master: 17bdfbef628a4fc3819023753ebe8c35 (127.12.12.3:34171)
I20260621 07:51:45.770735 13449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:51431
I20260621 07:51:45.777245 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.770286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:34251 (local address 127.12.12.3:55815)
0621 07:51:45.770592 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:45.770606 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:45.770704 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:45.770946 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:45.770949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:45.771178 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:45.771302 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.771309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.772311 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.772314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.772860 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.772867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.773286 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.773288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.773356 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.773816 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:45.773827 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:45.774149 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:45.775997 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:45.776000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:45.776002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:45.776210 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:45.776445 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:45.776447 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:45.776448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:45.776517 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:45.776820 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:45.776821 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:45.776896 (+ 75us) client_negotiation.cc:769] Sending connection context
0621 07:51:45.776973 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 07:51:45.777088 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":132,"threads_started":1}
I20260621 07:51:45.777325 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.770352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:55815 (local address 127.12.12.62:34251)
0621 07:51:45.770507 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:45.770526 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:45.770677 (+ 151us) server_negotiation.cc:407] Connection header received
0621 07:51:45.770788 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:45.770790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:45.770826 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:45.770929 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:45.771419 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.772174 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.772970 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.773186 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.773266 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.774294 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:45.774297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:45.774299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:45.774322 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:45.775876 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:45.776301 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:45.776303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:45.776304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:45.776375 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:45.776586 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:45.776588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:45.776589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:45.776748 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:45.776820 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:45.777097 (+ 277us) server_negotiation.cc:299] Negotiation successful
0621 07:51:45.777203 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 07:51:45.780918 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 07:51:45 dist-test-slave-lbpt krb5kdc[13406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 07:51:45 dist-test-slave-lbpt krb5kdc[13406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:45.806905 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.798261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.12.12.62:34251)
0621 07:51:45.798393 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:45.798397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:45.798409 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:51:45.798511 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:45.798514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:45.798610 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:45.798699 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:45.799633 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.800177 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.800972 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.801200 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.801295 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.803943 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:45.803946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:45.803948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:45.803975 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:45.805454 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:45.805843 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:45.805846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:45.805847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:45.805918 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:45.806135 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:45.806137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:45.806138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:45.806304 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:45.806399 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:45.806667 (+ 268us) server_negotiation.cc:299] Negotiation successful
0621 07:51:45.806781 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:51:45.809937 13449 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:51:45.812932 13449 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:51:45.827363 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.821839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:40541 (local address 127.0.0.1:33070)
0621 07:51:45.822644 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:45.822657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:45.822763 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:45.823185 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:45.823188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:45.823204 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:45.823429 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.823435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.824533 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.824537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.825746 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.825757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.826768 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.826771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.826892 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.826934 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 07:51:45.827038 (+ 104us) client_negotiation.cc:240] Negotiation successful
0621 07:51:45.827158 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":80,"thread_start_us":91,"threads_started":1}
I20260621 07:51:45.827395 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.822193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.12.12.2:40541)
0621 07:51:45.822502 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:45.822514 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:45.822760 (+ 246us) server_negotiation.cc:407] Connection header received
0621 07:51:45.822872 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:45.822877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:45.823039 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:45.823177 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:45.823557 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.824382 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.825889 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.826645 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.826828 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.827078 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:45.827134 (+ 56us) server_negotiation.cc:299] Negotiation successful
0621 07:51:45.827244 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":99,"threads_started":1}
I20260621 07:51:45.827850 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.821064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:36481 (local address 127.0.0.1:41196)
0621 07:51:45.821998 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:45.822051 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:45.822260 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:45.823196 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:45.823199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:45.823217 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:45.823429 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.823435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.824768 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.824770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.826139 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.826150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.827388 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.827393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.827491 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.827520 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 07:51:45.827594 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 07:51:45.827693 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":144,"thread_start_us":127,"threads_started":1}
I20260621 07:51:45.828336 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.821535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.12.12.3:34171)
0621 07:51:45.821933 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:45.821946 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:45.822491 (+ 545us) server_negotiation.cc:407] Connection header received
0621 07:51:45.822584 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:45.822589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:45.822852 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:45.823021 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:45.823299 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.824607 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.826513 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.827273 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.827372 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.827559 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:45.828116 (+ 557us) server_negotiation.cc:299] Negotiation successful
0621 07:51:45.828210 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":99,"threads_started":1}
I20260621 07:51:45.828394 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.821196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.12.12.1:36481)
0621 07:51:45.821743 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:45.821762 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:45.822137 (+ 375us) server_negotiation.cc:407] Connection header received
0621 07:51:45.822567 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:45.822575 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:45.822884 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:45.823011 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:45.823583 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.824655 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.826272 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.827100 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.827196 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.827324 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:45.828129 (+ 805us) server_negotiation.cc:299] Negotiation successful
0621 07:51:45.828240 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":106,"threads_started":1}
I20260621 07:51:45.828024 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.821126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:34171 (local address 127.0.0.1:60878)
0621 07:51:45.822342 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:45.822355 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:45.822479 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:45.823011 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:45.823017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:45.823038 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:45.823178 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.823186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.824762 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.824765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.826139 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.826150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.827620 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.827623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.827720 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.827741 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:51:45.827812 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 07:51:45.827883 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1157,"mutex_wait_us":78,"thread_start_us":102,"threads_started":1}
I20260621 07:51:45.830852 13728 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2bf2615f7d42e1b3f9f218abb2fa66 (DEFAULT_TABLE table=test-table [id=5038f03b573d4217ba352ce512da6256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:45.830886 13588 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2bf2615f7d42e1b3f9f218abb2fa66 (DEFAULT_TABLE table=test-table [id=5038f03b573d4217ba352ce512da6256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:45.831974 13728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2bf2615f7d42e1b3f9f218abb2fa66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:45.831723 13868 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2bf2615f7d42e1b3f9f218abb2fa66 (DEFAULT_TABLE table=test-table [id=5038f03b573d4217ba352ce512da6256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:45.832103 13588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2bf2615f7d42e1b3f9f218abb2fa66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:45.832583 13868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2bf2615f7d42e1b3f9f218abb2fa66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:45.838143 13957 tablet_bootstrap.cc:492] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8: Bootstrap starting.
I20260621 07:51:45.839907 13958 tablet_bootstrap.cc:492] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136: Bootstrap starting.
I20260621 07:51:45.840787 13957 tablet_bootstrap.cc:654] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:45.840989 13959 tablet_bootstrap.cc:492] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35: Bootstrap starting.
I20260621 07:51:45.841677 13957 log.cc:824] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:45.842274 13958 tablet_bootstrap.cc:654] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:45.842926 13959 tablet_bootstrap.cc:654] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:45.843211 13958 log.cc:824] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:45.843694 13959 log.cc:824] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:45.844151 13957 tablet_bootstrap.cc:492] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8: No bootstrap required, opened a new log
I20260621 07:51:45.844352 13957 ts_tablet_manager.cc:1397] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:51:45.845321 13958 tablet_bootstrap.cc:492] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136: No bootstrap required, opened a new log
I20260621 07:51:45.845415 13959 tablet_bootstrap.cc:492] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35: No bootstrap required, opened a new log
I20260621 07:51:45.845512 13958 ts_tablet_manager.cc:1397] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:51:45.845633 13959 ts_tablet_manager.cc:1397] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:51:45.848546 13959 raft_consensus.cc:348] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } }
I20260621 07:51:45.848816 13959 raft_consensus.cc:374] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:45.848898 13959 raft_consensus.cc:729] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17bdfbef628a4fc3819023753ebe8c35, State: Initialized, Role: FOLLOWER
I20260621 07:51:45.848742 13957 raft_consensus.cc:348] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } }
I20260621 07:51:45.849038 13957 raft_consensus.cc:374] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:45.849135 13957 raft_consensus.cc:729] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb16501b843492eadf65978e1008de8, State: Initialized, Role: FOLLOWER
I20260621 07:51:45.849282 13958 raft_consensus.cc:348] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } }
I20260621 07:51:45.849421 13959 consensus_queue.cc:260] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } }
I20260621 07:51:45.849575 13958 raft_consensus.cc:374] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:45.849649 13958 raft_consensus.cc:729] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ace1cfa354946b9b9e5196ac0de1136, State: Initialized, Role: FOLLOWER
I20260621 07:51:45.849658 13957 consensus_queue.cc:260] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } }
I20260621 07:51:45.850119 13958 consensus_queue.cc:260] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } }
I20260621 07:51:45.850669 13957 ts_tablet_manager.cc:1428] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260621 07:51:45.850656 13959 ts_tablet_manager.cc:1428] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:51:45.850909 13934 heartbeater.cc:503] Master 127.12.12.62:34251 was elected leader, sending a full tablet report...
I20260621 07:51:45.851080 13958 ts_tablet_manager.cc:1428] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260621 07:51:45.851102 13794 heartbeater.cc:503] Master 127.12.12.62:34251 was elected leader, sending a full tablet report...
I20260621 07:51:45.852101 13654 heartbeater.cc:503] Master 127.12.12.62:34251 was elected leader, sending a full tablet report...
I20260621 07:51:45.871786 13964 raft_consensus.cc:482] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:51:45.871953 13964 raft_consensus.cc:504] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } }
I20260621 07:51:45.873005 13964 leader_election.cc:288] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17bdfbef628a4fc3819023753ebe8c35 (127.12.12.3:34171), 7ace1cfa354946b9b9e5196ac0de1136 (127.12.12.2:40541)
I20260621 07:51:45.878443 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.873322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:42199 (local address 127.12.12.3:34171)
0621 07:51:45.873446 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:45.873451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:45.873824 (+ 373us) server_negotiation.cc:407] Connection header received
0621 07:51:45.873909 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:45.873911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:45.873965 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:45.874050 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:45.874496 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.875306 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.877163 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.877767 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.877877 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.877939 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:45.878212 (+ 273us) server_negotiation.cc:299] Negotiation successful
0621 07:51:45.878291 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:51:45.878443 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.873361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:50453 (local address 127.12.12.2:40541)
0621 07:51:45.873475 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:45.873479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:45.873714 (+ 235us) server_negotiation.cc:407] Connection header received
0621 07:51:45.873875 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:45.873877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:45.873933 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:45.874023 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:45.874511 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.875306 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.877170 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.877767 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.877877 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.877939 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:45.878228 (+ 289us) server_negotiation.cc:299] Negotiation successful
0621 07:51:45.878296 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:51:45.878525 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.873224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:34171 (local address 127.12.12.1:42199)
0621 07:51:45.873701 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:45.873714 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:45.873822 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:45.874060 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:45.874063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:45.874106 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:45.874356 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.874364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.875420 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.875423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.877028 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.877040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.877897 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.877901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.878055 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.878095 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 07:51:45.878177 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 07:51:45.878297 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":299,"spinlock_wait_cycles":30720,"thread_start_us":100,"threads_started":1}
I20260621 07:51:45.878527 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.873224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:40541 (local address 127.12.12.1:50453)
0621 07:51:45.873610 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:45.873626 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:45.873748 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:45.874060 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:45.874063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:45.874107 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:45.874356 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.874364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.875420 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.875423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.877027 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:45.877040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.877897 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.877901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:45.878094 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.878112 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 07:51:45.878177 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 07:51:45.878295 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":268,"spinlock_wait_cycles":7424,"thread_start_us":122,"threads_started":1}
I20260621 07:51:45.879199 13888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2bf2615f7d42e1b3f9f218abb2fa66" candidate_uuid: "eeb16501b843492eadf65978e1008de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17bdfbef628a4fc3819023753ebe8c35" is_pre_election: true
I20260621 07:51:45.879247 13748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2bf2615f7d42e1b3f9f218abb2fa66" candidate_uuid: "eeb16501b843492eadf65978e1008de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ace1cfa354946b9b9e5196ac0de1136" is_pre_election: true
I20260621 07:51:45.879524 13748 raft_consensus.cc:2454] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb16501b843492eadf65978e1008de8 in term 0.
I20260621 07:51:45.879525 13888 raft_consensus.cc:2454] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb16501b843492eadf65978e1008de8 in term 0.
I20260621 07:51:45.880036 13541 leader_election.cc:302] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ace1cfa354946b9b9e5196ac0de1136, eeb16501b843492eadf65978e1008de8; no voters:
I20260621 07:51:45.880314 13964 raft_consensus.cc:2781] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:51:45.880401 13964 raft_consensus.cc:482] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:51:45.880445 13964 raft_consensus.cc:3037] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:45.881624 13964 raft_consensus.cc:504] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } }
I20260621 07:51:45.881973 13964 leader_election.cc:288] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [CANDIDATE]: Term 1 election: Requested vote from peers 17bdfbef628a4fc3819023753ebe8c35 (127.12.12.3:34171), 7ace1cfa354946b9b9e5196ac0de1136 (127.12.12.2:40541)
I20260621 07:51:45.882246 13888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2bf2615f7d42e1b3f9f218abb2fa66" candidate_uuid: "eeb16501b843492eadf65978e1008de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17bdfbef628a4fc3819023753ebe8c35"
I20260621 07:51:45.882308 13748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2bf2615f7d42e1b3f9f218abb2fa66" candidate_uuid: "eeb16501b843492eadf65978e1008de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ace1cfa354946b9b9e5196ac0de1136"
I20260621 07:51:45.882386 13888 raft_consensus.cc:3037] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:45.882453 13748 raft_consensus.cc:3037] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:45.883481 13888 raft_consensus.cc:2454] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb16501b843492eadf65978e1008de8 in term 1.
I20260621 07:51:45.883798 13542 leader_election.cc:302] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17bdfbef628a4fc3819023753ebe8c35, eeb16501b843492eadf65978e1008de8; no voters:
I20260621 07:51:45.883817 13748 raft_consensus.cc:2454] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb16501b843492eadf65978e1008de8 in term 1.
I20260621 07:51:45.883996 13964 raft_consensus.cc:2781] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:45.884294 13964 raft_consensus.cc:686] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [term 1 LEADER]: Becoming Leader. State: Replica: eeb16501b843492eadf65978e1008de8, State: Running, Role: LEADER
I20260621 07:51:45.884619 13964 consensus_queue.cc:237] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } }
I20260621 07:51:45.887681 13450 catalog_manager.cc:5495] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 reported cstate change: term changed from 0 to 1, leader changed from <none> to eeb16501b843492eadf65978e1008de8 (127.12.12.1). New cstate: current_term: 1 leader_uuid: "eeb16501b843492eadf65978e1008de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } health_report { overall_health: UNKNOWN } } }
I20260621 07:51:45.910579 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:45.907118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.12.12.1:36481)
0621 07:51:45.907260 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:45.907264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:45.907278 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:51:45.907438 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:45.907441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:45.907498 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:45.907585 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:45.907833 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.908349 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.909198 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:45.909422 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:45.909500 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:45.909770 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:45.910192 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:45.910277 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:45.910349 (+ 72us) server_negotiation.cc:299] Negotiation successful
0621 07:51:45.910396 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 07:51:45.916333 13888 raft_consensus.cc:1261] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35 [term 1 FOLLOWER]: Refusing update from remote peer eeb16501b843492eadf65978e1008de8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:45.916375 13748 raft_consensus.cc:1261] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136 [term 1 FOLLOWER]: Refusing update from remote peer eeb16501b843492eadf65978e1008de8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:45.917083 13968 consensus_queue.cc:1035] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:45.917284 13964 consensus_queue.cc:1035] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:45.925566 13973 mvcc.cc:205] Tried to move back new op lower bound from 7299187941026058240 to 7299187940905963520. Current Snapshot: MvccSnapshot[applied={T|T < 7299187941026058240}]
I20260621 07:51:45.936691 13451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:45.936872 13451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:45.939198 13451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e: Sending DeleteTablet for 3 replicas of tablet 9b2bf2615f7d42e1b3f9f218abb2fa66
I20260621 07:51:45.939975 13868 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2bf2615f7d42e1b3f9f218abb2fa66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:45 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:60878
I20260621 07:51:45.939989 13588 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2bf2615f7d42e1b3f9f218abb2fa66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:45 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:41196
I20260621 07:51:45.940032 13728 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2bf2615f7d42e1b3f9f218abb2fa66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:45 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:33070
I20260621 07:51:45.940487 13984 tablet_replica.cc:330] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8: stopping tablet replica
I20260621 07:51:45.940652 13982 tablet_replica.cc:330] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35: stopping tablet replica
I20260621 07:51:45.940878 13983 tablet_replica.cc:330] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136: stopping tablet replica
I20260621 07:51:45.940927 13984 raft_consensus.cc:2229] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:51:45.941241 13983 raft_consensus.cc:2229] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:45.941447 13984 raft_consensus.cc:2258] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:45.941545 13983 raft_consensus.cc:2258] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:45.941488 13451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:51:45.942143 13451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:51:45.942284 13982 raft_consensus.cc:2229] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:45.942685 13982 raft_consensus.cc:2258] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:45.943395 13984 ts_tablet_manager.cc:1905] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:45.944686 13982 ts_tablet_manager.cc:1905] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:45.946172 13984 ts_tablet_manager.cc:1918] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:51:45.946337 13984 log.cc:1196] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/wal/wals/9b2bf2615f7d42e1b3f9f218abb2fa66
I20260621 07:51:45.946754 13984 ts_tablet_manager.cc:1939] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P eeb16501b843492eadf65978e1008de8: Deleting consensus metadata
I20260621 07:51:45.947351 13868 tablet_service.cc:1467] Processing CreateTablet for tablet 490c1633c2c3472b9f28bb974ccb1e6f (DEFAULT_TABLE table=test-table [id=ef3aa6713b934f10b2b65de168966efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:45.947351 13588 tablet_service.cc:1467] Processing CreateTablet for tablet 490c1633c2c3472b9f28bb974ccb1e6f (DEFAULT_TABLE table=test-table [id=ef3aa6713b934f10b2b65de168966efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:45.947587 13588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 490c1633c2c3472b9f28bb974ccb1e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:45.947587 13868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 490c1633c2c3472b9f28bb974ccb1e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:45.949676 13957 tablet_bootstrap.cc:492] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8: Bootstrap starting.
I20260621 07:51:45.949978 13437 catalog_manager.cc:4841] TS eeb16501b843492eadf65978e1008de8 (127.12.12.1:36481): tablet 9b2bf2615f7d42e1b3f9f218abb2fa66 (table test-table [id=5038f03b573d4217ba352ce512da6256]) successfully deleted
I20260621 07:51:45.950479 13982 ts_tablet_manager.cc:1918] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:51:45.950484 13728 tablet_service.cc:1467] Processing CreateTablet for tablet 490c1633c2c3472b9f28bb974ccb1e6f (DEFAULT_TABLE table=test-table [id=ef3aa6713b934f10b2b65de168966efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:45.950587 13982 log.cc:1196] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/9b2bf2615f7d42e1b3f9f218abb2fa66
I20260621 07:51:45.950623 13957 tablet_bootstrap.cc:654] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:45.950789 13728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 490c1633c2c3472b9f28bb974ccb1e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:45.950894 13982 ts_tablet_manager.cc:1939] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 17bdfbef628a4fc3819023753ebe8c35: Deleting consensus metadata
I20260621 07:51:45.951205 13959 tablet_bootstrap.cc:492] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35: Bootstrap starting.
I20260621 07:51:45.951997 13438 catalog_manager.cc:4841] TS 17bdfbef628a4fc3819023753ebe8c35 (127.12.12.3:34171): tablet 9b2bf2615f7d42e1b3f9f218abb2fa66 (table test-table [id=5038f03b573d4217ba352ce512da6256]) successfully deleted
I20260621 07:51:45.952075 13959 tablet_bootstrap.cc:654] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:45.953133 13958 tablet_bootstrap.cc:492] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136: Bootstrap starting.
I20260621 07:51:45.953246 13959 tablet_bootstrap.cc:492] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35: No bootstrap required, opened a new log
I20260621 07:51:45.953323 13959 ts_tablet_manager.cc:1397] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:51:45.953522 13957 tablet_bootstrap.cc:492] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8: No bootstrap required, opened a new log
I20260621 07:51:45.953603 13957 ts_tablet_manager.cc:1397] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260621 07:51:45.953763 13959 raft_consensus.cc:348] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:45.953881 13959 raft_consensus.cc:374] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:45.953914 13959 raft_consensus.cc:729] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17bdfbef628a4fc3819023753ebe8c35, State: Initialized, Role: FOLLOWER
I20260621 07:51:45.954075 13958 tablet_bootstrap.cc:654] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:45.954048 13959 consensus_queue.cc:260] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:45.954037 13957 raft_consensus.cc:348] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:45.954142 13957 raft_consensus.cc:374] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:45.954175 13957 raft_consensus.cc:729] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb16501b843492eadf65978e1008de8, State: Initialized, Role: FOLLOWER
I20260621 07:51:45.954313 13959 ts_tablet_manager.cc:1428] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:51:45.955004 13983 ts_tablet_manager.cc:1905] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:45.954317 13957 consensus_queue.cc:260] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:45.955351 13957 ts_tablet_manager.cc:1428] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:51:45.957487 13983 ts_tablet_manager.cc:1918] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:51:45.957583 13983 log.cc:1196] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/9b2bf2615f7d42e1b3f9f218abb2fa66
I20260621 07:51:45.957834 13958 tablet_bootstrap.cc:492] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136: No bootstrap required, opened a new log
I20260621 07:51:45.957924 13958 ts_tablet_manager.cc:1397] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260621 07:51:45.957940 13983 ts_tablet_manager.cc:1939] T 9b2bf2615f7d42e1b3f9f218abb2fa66 P 7ace1cfa354946b9b9e5196ac0de1136: Deleting consensus metadata
I20260621 07:51:45.958370 13958 raft_consensus.cc:348] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:45.958510 13958 raft_consensus.cc:374] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:45.958551 13958 raft_consensus.cc:729] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ace1cfa354946b9b9e5196ac0de1136, State: Initialized, Role: FOLLOWER
I20260621 07:51:45.958671 13958 consensus_queue.cc:260] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:45.958895 13958 ts_tablet_manager.cc:1428] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260621 07:51:45.959144 13437 catalog_manager.cc:4841] TS 7ace1cfa354946b9b9e5196ac0de1136 (127.12.12.2:40541): tablet 9b2bf2615f7d42e1b3f9f218abb2fa66 (table test-table [id=5038f03b573d4217ba352ce512da6256]) successfully deleted
W20260621 07:51:46.015071 13935 tablet.cc:2369] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:51:46.024303 13965 raft_consensus.cc:482] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:51:46.024485 13965 raft_consensus.cc:504] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:46.025714 13964 raft_consensus.cc:482] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:51:46.025799 13965 leader_election.cc:288] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeb16501b843492eadf65978e1008de8 (127.12.12.1:36481), 17bdfbef628a4fc3819023753ebe8c35 (127.12.12.3:34171)
I20260621 07:51:46.025861 13964 raft_consensus.cc:504] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:46.026275 13964 leader_election.cc:288] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ace1cfa354946b9b9e5196ac0de1136 (127.12.12.2:40541), 17bdfbef628a4fc3819023753ebe8c35 (127.12.12.3:34171)
I20260621 07:51:46.027045 13748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490c1633c2c3472b9f28bb974ccb1e6f" candidate_uuid: "eeb16501b843492eadf65978e1008de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ace1cfa354946b9b9e5196ac0de1136" is_pre_election: true
I20260621 07:51:46.027208 13748 raft_consensus.cc:2454] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb16501b843492eadf65978e1008de8 in term 0.
I20260621 07:51:46.027554 13541 leader_election.cc:302] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ace1cfa354946b9b9e5196ac0de1136, eeb16501b843492eadf65978e1008de8; no voters:
I20260621 07:51:46.027720 13964 raft_consensus.cc:2781] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:51:46.027768 13964 raft_consensus.cc:482] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:51:46.027805 13964 raft_consensus.cc:3037] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:46.029012 13964 raft_consensus.cc:504] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:46.029333 13964 leader_election.cc:288] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [CANDIDATE]: Term 1 election: Requested vote from peers 7ace1cfa354946b9b9e5196ac0de1136 (127.12.12.2:40541), 17bdfbef628a4fc3819023753ebe8c35 (127.12.12.3:34171)
I20260621 07:51:46.029883 13888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490c1633c2c3472b9f28bb974ccb1e6f" candidate_uuid: "eeb16501b843492eadf65978e1008de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17bdfbef628a4fc3819023753ebe8c35" is_pre_election: true
I20260621 07:51:46.030027 13888 raft_consensus.cc:2454] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb16501b843492eadf65978e1008de8 in term 0.
I20260621 07:51:46.030629 13887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490c1633c2c3472b9f28bb974ccb1e6f" candidate_uuid: "eeb16501b843492eadf65978e1008de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17bdfbef628a4fc3819023753ebe8c35"
I20260621 07:51:46.030755 13887 raft_consensus.cc:3037] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:46.031193 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:46.026038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:36481 (local address 127.12.12.2:38551)
0621 07:51:46.026334 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:46.026352 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:46.026472 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:46.026908 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:46.026911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:46.026929 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:46.027040 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:46.027046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.028059 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.028062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:46.029763 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:46.029776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.030773 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.030776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:46.030874 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:46.030906 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 07:51:46.030986 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 07:51:46.031065 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":438}
I20260621 07:51:46.031271 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:46.026652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:41379 (local address 127.12.12.3:34171)
0621 07:51:46.026776 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:46.026783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:46.026844 (+ 61us) server_negotiation.cc:407] Connection header received
0621 07:51:46.027023 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:46.027026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:46.027084 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:46.027137 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:46.027555 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.028262 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.029727 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.030353 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.030470 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:46.030565 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:46.031085 (+ 520us) server_negotiation.cc:299] Negotiation successful
0621 07:51:46.031164 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 07:51:46.031679 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:46.026484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:38551 (local address 127.12.12.1:36481)
0621 07:51:46.026602 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:46.026606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:46.026620 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:51:46.026668 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:46.026671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:46.026723 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:46.026796 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:46.027164 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.027942 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.029908 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.030567 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.030660 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:46.030719 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:46.031456 (+ 737us) server_negotiation.cc:299] Negotiation successful
0621 07:51:46.031532 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 07:51:46.031708 13988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:46.026106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:34171 (local address 127.12.12.2:41379)
0621 07:51:46.026757 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:46.026771 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:46.026904 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:46.027277 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:46.027279 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:46.027295 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:46.027431 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:46.027437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.028400 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.028403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:46.029586 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:46.029597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.030807 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.030812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:46.030921 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:46.030959 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 07:51:46.031382 (+ 423us) client_negotiation.cc:240] Negotiation successful
0621 07:51:46.031458 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":5,"thread_start_us":97,"threads_started":1}
I20260621 07:51:46.032081 13608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490c1633c2c3472b9f28bb974ccb1e6f" candidate_uuid: "7ace1cfa354946b9b9e5196ac0de1136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb16501b843492eadf65978e1008de8" is_pre_election: true
I20260621 07:51:46.032315 13608 raft_consensus.cc:2379] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ace1cfa354946b9b9e5196ac0de1136 in current term 1: Already voted for candidate eeb16501b843492eadf65978e1008de8 in this term.
I20260621 07:51:46.032341 13748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490c1633c2c3472b9f28bb974ccb1e6f" candidate_uuid: "eeb16501b843492eadf65978e1008de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ace1cfa354946b9b9e5196ac0de1136"
I20260621 07:51:46.032452 13748 raft_consensus.cc:3037] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:46.033375 13888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490c1633c2c3472b9f28bb974ccb1e6f" candidate_uuid: "7ace1cfa354946b9b9e5196ac0de1136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17bdfbef628a4fc3819023753ebe8c35" is_pre_election: true
I20260621 07:51:46.033684 13748 raft_consensus.cc:2454] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb16501b843492eadf65978e1008de8 in term 1.
I20260621 07:51:46.034004 13541 leader_election.cc:302] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ace1cfa354946b9b9e5196ac0de1136, eeb16501b843492eadf65978e1008de8; no voters:
I20260621 07:51:46.034238 13964 raft_consensus.cc:2781] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:46.034165 13682 leader_election.cc:302] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ace1cfa354946b9b9e5196ac0de1136; no voters: 17bdfbef628a4fc3819023753ebe8c35, eeb16501b843492eadf65978e1008de8
I20260621 07:51:46.034327 13964 raft_consensus.cc:686] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [term 1 LEADER]: Becoming Leader. State: Replica: eeb16501b843492eadf65978e1008de8, State: Running, Role: LEADER
I20260621 07:51:46.034502 13964 consensus_queue.cc:237] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:46.036273 13887 raft_consensus.cc:2454] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb16501b843492eadf65978e1008de8 in term 1.
I20260621 07:51:46.036788 13965 raft_consensus.cc:2726] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260621 07:51:46.037315 13795 tablet.cc:2369] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:51:46.037586 13449 catalog_manager.cc:5495] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 reported cstate change: term changed from 0 to 1, leader changed from <none> to eeb16501b843492eadf65978e1008de8 (127.12.12.1). New cstate: current_term: 1 leader_uuid: "eeb16501b843492eadf65978e1008de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } health_report { overall_health: UNKNOWN } } }
W20260621 07:51:46.069657 13655 tablet.cc:2369] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 07:51:46 dist-test-slave-lbpt krb5kdc[13406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028304, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:46.085733 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:46.076178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34942 (local address 127.12.12.62:34251)
0621 07:51:46.076327 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:46.076332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:46.076593 (+ 261us) server_negotiation.cc:407] Connection header received
0621 07:51:46.076763 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:46.076765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:46.076803 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:46.076873 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:46.077764 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.078273 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.079428 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.079630 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.079708 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:46.082474 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:46.082477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:46.082479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:46.082500 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:46.083967 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:46.084521 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:46.084524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:46.084525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:46.084598 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:46.084867 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:46.084869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:46.084870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:46.085038 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:46.085112 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:46.085435 (+ 323us) server_negotiation.cc:299] Negotiation successful
0621 07:51:46.085587 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":55}
I20260621 07:51:46.085767 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:46.076060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:34251 (local address 127.0.0.1:34942)
0621 07:51:46.076488 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:46.076503 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:46.076641 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:46.076937 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:46.076940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:46.077495 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:46.077617 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:46.077625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.078389 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.078393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:46.079310 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:46.079321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.079755 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.079759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:46.079852 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:46.080456 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:46.080471 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:46.082290 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:46.084082 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:46.084093 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:46.084100 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:46.084419 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:46.084687 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:46.084690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:46.084692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:46.084783 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:46.085134 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:46.085141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:46.085264 (+ 123us) client_negotiation.cc:769] Sending connection context
0621 07:51:46.085378 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 07:51:46.085530 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":142,"threads_started":1}
I20260621 07:51:46.090148 13449 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:34942:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 07:51:46.091027 13449 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:51:46.096130 13728 tablet_service.cc:1467] Processing CreateTablet for tablet a6f980a8bb1849a996f86187926b698c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b8200491bd34deca47d658622658542]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:51:46.096131 13588 tablet_service.cc:1467] Processing CreateTablet for tablet a6f980a8bb1849a996f86187926b698c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b8200491bd34deca47d658622658542]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:51:46.096247 13868 tablet_service.cc:1467] Processing CreateTablet for tablet a6f980a8bb1849a996f86187926b698c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b8200491bd34deca47d658622658542]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:51:46.096556 13728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6f980a8bb1849a996f86187926b698c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:46.096563 13868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6f980a8bb1849a996f86187926b698c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:46.096571 13588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6f980a8bb1849a996f86187926b698c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:46.099478 13959 tablet_bootstrap.cc:492] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35: Bootstrap starting.
I20260621 07:51:46.099695 13957 tablet_bootstrap.cc:492] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8: Bootstrap starting.
I20260621 07:51:46.099766 13958 tablet_bootstrap.cc:492] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136: Bootstrap starting.
I20260621 07:51:46.100497 13959 tablet_bootstrap.cc:654] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:46.100497 13957 tablet_bootstrap.cc:654] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:46.100849 13958 tablet_bootstrap.cc:654] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:46.102523 13957 tablet_bootstrap.cc:492] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8: No bootstrap required, opened a new log
I20260621 07:51:46.102619 13957 ts_tablet_manager.cc:1397] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 07:51:46.102746 13958 tablet_bootstrap.cc:492] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136: No bootstrap required, opened a new log
I20260621 07:51:46.102823 13958 ts_tablet_manager.cc:1397] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260621 07:51:46.103044 13957 raft_consensus.cc:348] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:46.103186 13957 raft_consensus.cc:374] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:46.103233 13957 raft_consensus.cc:729] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb16501b843492eadf65978e1008de8, State: Initialized, Role: FOLLOWER
I20260621 07:51:46.103279 13958 raft_consensus.cc:348] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:46.103396 13958 raft_consensus.cc:374] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:46.103430 13958 raft_consensus.cc:729] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ace1cfa354946b9b9e5196ac0de1136, State: Initialized, Role: FOLLOWER
I20260621 07:51:46.103377 13957 consensus_queue.cc:260] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:46.103540 13958 consensus_queue.cc:260] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:46.103691 13957 ts_tablet_manager.cc:1428] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:51:46.103772 13958 ts_tablet_manager.cc:1428] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:51:46.103926 13976 tablet_replica.cc:439] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } }
I20260621 07:51:46.103971 13965 tablet_replica.cc:439] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } }
I20260621 07:51:46.104061 13976 tablet_replica.cc:442] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:51:46.104102 13965 tablet_replica.cc:442] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:51:46.105547 13959 tablet_bootstrap.cc:492] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35: No bootstrap required, opened a new log
I20260621 07:51:46.105646 13959 ts_tablet_manager.cc:1397] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260621 07:51:46.106091 13959 raft_consensus.cc:348] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:46.106208 13959 raft_consensus.cc:374] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:46.106254 13959 raft_consensus.cc:729] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17bdfbef628a4fc3819023753ebe8c35, State: Initialized, Role: FOLLOWER
I20260621 07:51:46.106389 13959 consensus_queue.cc:260] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:46.106724 13959 ts_tablet_manager.cc:1428] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:51:46.107081 13963 tablet_replica.cc:439] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } }
I20260621 07:51:46.107221 13963 tablet_replica.cc:442] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:51:46.165727 13976 raft_consensus.cc:482] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:51:46.165922 13976 raft_consensus.cc:504] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:46.166436 13976 leader_election.cc:288] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ace1cfa354946b9b9e5196ac0de1136 (127.12.12.2:40541), 17bdfbef628a4fc3819023753ebe8c35 (127.12.12.3:34171)
I20260621 07:51:46.166932 13887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f980a8bb1849a996f86187926b698c" candidate_uuid: "eeb16501b843492eadf65978e1008de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17bdfbef628a4fc3819023753ebe8c35" is_pre_election: true
I20260621 07:51:46.166932 13748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f980a8bb1849a996f86187926b698c" candidate_uuid: "eeb16501b843492eadf65978e1008de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ace1cfa354946b9b9e5196ac0de1136" is_pre_election: true
I20260621 07:51:46.167099 13748 raft_consensus.cc:2454] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb16501b843492eadf65978e1008de8 in term 0.
I20260621 07:51:46.167099 13887 raft_consensus.cc:2454] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb16501b843492eadf65978e1008de8 in term 0.
I20260621 07:51:46.167392 13541 leader_election.cc:302] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ace1cfa354946b9b9e5196ac0de1136, eeb16501b843492eadf65978e1008de8; no voters:
I20260621 07:51:46.167578 13976 raft_consensus.cc:2781] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:51:46.167639 13976 raft_consensus.cc:482] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:51:46.167667 13976 raft_consensus.cc:3037] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:46.168819 13976 raft_consensus.cc:504] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:46.169189 13976 leader_election.cc:288] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [CANDIDATE]: Term 1 election: Requested vote from peers 7ace1cfa354946b9b9e5196ac0de1136 (127.12.12.2:40541), 17bdfbef628a4fc3819023753ebe8c35 (127.12.12.3:34171)
I20260621 07:51:46.169555 13748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f980a8bb1849a996f86187926b698c" candidate_uuid: "eeb16501b843492eadf65978e1008de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ace1cfa354946b9b9e5196ac0de1136"
I20260621 07:51:46.169600 13887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f980a8bb1849a996f86187926b698c" candidate_uuid: "eeb16501b843492eadf65978e1008de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17bdfbef628a4fc3819023753ebe8c35"
I20260621 07:51:46.169695 13748 raft_consensus.cc:3037] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:46.169726 13887 raft_consensus.cc:3037] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:46.170864 13887 raft_consensus.cc:2454] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb16501b843492eadf65978e1008de8 in term 1.
I20260621 07:51:46.170864 13748 raft_consensus.cc:2454] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb16501b843492eadf65978e1008de8 in term 1.
I20260621 07:51:46.171185 13541 leader_election.cc:302] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ace1cfa354946b9b9e5196ac0de1136, eeb16501b843492eadf65978e1008de8; no voters:
I20260621 07:51:46.171417 13976 raft_consensus.cc:2781] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:46.171538 13976 raft_consensus.cc:686] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [term 1 LEADER]: Becoming Leader. State: Replica: eeb16501b843492eadf65978e1008de8, State: Running, Role: LEADER
I20260621 07:51:46.171679 13976 consensus_queue.cc:237] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } }
I20260621 07:51:46.172446 13964 tablet_replica.cc:439] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8: TxnStatusTablet state changed. Reason: New leader eeb16501b843492eadf65978e1008de8. Latest consensus state: current_term: 1 leader_uuid: "eeb16501b843492eadf65978e1008de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } }
I20260621 07:51:46.172561 13964 tablet_replica.cc:442] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:51:46.172998 14000 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 07:51:46.173182 14000 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 07:51:46.173864 13449 catalog_manager.cc:5495] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 reported cstate change: term changed from 0 to 1, leader changed from <none> to eeb16501b843492eadf65978e1008de8 (127.12.12.1). New cstate: current_term: 1 leader_uuid: "eeb16501b843492eadf65978e1008de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } health_report { overall_health: UNKNOWN } } }
I20260621 07:51:46.190685 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:46.186220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:36481 (local address 127.0.0.1:41218)
0621 07:51:46.186404 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:46.186446 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:46.186590 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:46.186979 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:46.186982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:46.186992 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:51:46.187098 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:46.187104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.188238 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.188242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:46.189318 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:46.189329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.189836 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.189840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:46.189946 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:46.189993 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:51:46.190318 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:51:46.190337 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 07:51:46.190400 (+ 63us) client_negotiation.cc:240] Negotiation successful
0621 07:51:46.190548 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260621 07:51:46.190685 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:46.186545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.12.12.1:36481)
0621 07:51:46.186655 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:46.186660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:46.186673 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:51:46.186723 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:46.186726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:46.186780 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:46.186854 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:46.187235 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.188080 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.189446 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.189695 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.189816 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:46.190142 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:46.190234 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:46.190314 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:46.190455 (+ 141us) server_negotiation.cc:299] Negotiation successful
0621 07:51:46.190530 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 07:51:46.193457 13887 raft_consensus.cc:1261] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35 [term 1 FOLLOWER]: Refusing update from remote peer eeb16501b843492eadf65978e1008de8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:46.193457 13748 raft_consensus.cc:1261] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136 [term 1 FOLLOWER]: Refusing update from remote peer eeb16501b843492eadf65978e1008de8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:46.193845 13964 consensus_queue.cc:1035] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:46.193977 13976 consensus_queue.cc:1035] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:46.195335 13965 tablet_replica.cc:439] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136: TxnStatusTablet state changed. Reason: New leader eeb16501b843492eadf65978e1008de8. Latest consensus state: current_term: 1 leader_uuid: "eeb16501b843492eadf65978e1008de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } }
I20260621 07:51:46.195454 13965 tablet_replica.cc:442] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:51:46.195749 13963 tablet_replica.cc:439] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35: TxnStatusTablet state changed. Reason: New leader eeb16501b843492eadf65978e1008de8. Latest consensus state: current_term: 1 leader_uuid: "eeb16501b843492eadf65978e1008de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } }
I20260621 07:51:46.195861 13963 tablet_replica.cc:442] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:51:46.197412 13964 tablet_replica.cc:439] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eeb16501b843492eadf65978e1008de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } }
I20260621 07:51:46.197530 13964 tablet_replica.cc:442] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:51:46.197669 13971 mvcc.cc:205] Tried to move back new op lower bound from 7299187942163755008 to 7299187942080589824. Current Snapshot: MvccSnapshot[applied={T|T < 7299187942163755008 or (T in {7299187942163755008})}]
I20260621 07:51:46.198731 13972 mvcc.cc:205] Tried to move back new op lower bound from 7299187942163755008 to 7299187942080589824. Current Snapshot: MvccSnapshot[applied={T|T < 7299187942163755008}]
I20260621 07:51:46.198786 13980 tablet_replica.cc:439] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eeb16501b843492eadf65978e1008de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } }
I20260621 07:51:46.198891 13980 tablet_replica.cc:442] T a6f980a8bb1849a996f86187926b698c P eeb16501b843492eadf65978e1008de8: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:51:46.198915 13963 tablet_replica.cc:439] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eeb16501b843492eadf65978e1008de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } }
I20260621 07:51:46.199012 13963 tablet_replica.cc:442] T a6f980a8bb1849a996f86187926b698c P 17bdfbef628a4fc3819023753ebe8c35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:51:46.199901 13965 tablet_replica.cc:439] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eeb16501b843492eadf65978e1008de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb16501b843492eadf65978e1008de8" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36481 } } peers { permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 } } peers { permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 } } }
I20260621 07:51:46.200009 13965 tablet_replica.cc:442] T a6f980a8bb1849a996f86187926b698c P 7ace1cfa354946b9b9e5196ac0de1136: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:51:46.212253 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:46.208563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:45763 (local address 127.12.12.1:36481)
0621 07:51:46.208692 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:46.208697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:46.208934 (+ 237us) server_negotiation.cc:407] Connection header received
0621 07:51:46.209057 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:46.209060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:46.209104 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:46.209175 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:46.209449 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.209952 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.211180 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.211387 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.211466 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:46.211779 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:46.211848 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:46.211914 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:46.212025 (+ 111us) server_negotiation.cc:299] Negotiation successful
0621 07:51:46.212106 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":50}
I20260621 07:51:46.212289 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:46.208472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:36481 (local address 127.12.12.1:45763)
0621 07:51:46.208847 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:46.208861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:46.208967 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:46.209194 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:46.209197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:46.209204 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:51:46.209355 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:46.209362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.210050 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.210053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:46.211057 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:46.211068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.211492 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.211495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:46.211581 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:46.211630 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:51:46.211936 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:51:46.211942 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:51:46.212006 (+ 64us) client_negotiation.cc:240] Negotiation successful
0621 07:51:46.212107 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":18,"thread_start_us":101,"threads_started":1}
I20260621 07:51:46.218750 13887 raft_consensus.cc:1261] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35 [term 1 FOLLOWER]: Refusing update from remote peer eeb16501b843492eadf65978e1008de8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:46.218760 13748 raft_consensus.cc:1261] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136 [term 1 FOLLOWER]: Refusing update from remote peer eeb16501b843492eadf65978e1008de8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:46.219158 13980 consensus_queue.cc:1035] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ace1cfa354946b9b9e5196ac0de1136" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:46.219302 13976 consensus_queue.cc:1035] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17bdfbef628a4fc3819023753ebe8c35" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:46.260375 13451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:46.260569 13451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:46.262729 13451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8332ae99cfd4cd9a1d34e09eb12705e: Sending DeleteTablet for 3 replicas of tablet 490c1633c2c3472b9f28bb974ccb1e6f
I20260621 07:51:46.263298 13588 tablet_service.cc:1514] Processing DeleteTablet for tablet 490c1633c2c3472b9f28bb974ccb1e6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:46 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:41196
I20260621 07:51:46.263667 13728 tablet_service.cc:1514] Processing DeleteTablet for tablet 490c1633c2c3472b9f28bb974ccb1e6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:46 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:33070
I20260621 07:51:46.263773 13868 tablet_service.cc:1514] Processing DeleteTablet for tablet 490c1633c2c3472b9f28bb974ccb1e6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:46 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:60878
I20260621 07:51:46.263842 13983 tablet_replica.cc:330] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136: stopping tablet replica
I20260621 07:51:46.263952 13982 tablet_replica.cc:330] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35: stopping tablet replica
I20260621 07:51:46.263952 13983 raft_consensus.cc:2229] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:46.264063 13983 raft_consensus.cc:2258] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:46.264061 13982 raft_consensus.cc:2229] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:46.264182 13982 raft_consensus.cc:2258] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:46.264453 13984 tablet_replica.cc:330] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8: stopping tablet replica
I20260621 07:51:46.264569 13984 raft_consensus.cc:2229] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:51:46.264752 13984 raft_consensus.cc:2258] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:46.265550 13984 ts_tablet_manager.cc:1905] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:46.265569 13982 ts_tablet_manager.cc:1905] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:46.268298 13982 ts_tablet_manager.cc:1918] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:51:46.268391 13982 log.cc:1196] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/490c1633c2c3472b9f28bb974ccb1e6f
I20260621 07:51:46.268580 13984 ts_tablet_manager.cc:1918] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:51:46.268661 13984 log.cc:1196] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-0/wal/wals/490c1633c2c3472b9f28bb974ccb1e6f
I20260621 07:51:46.268630 13982 ts_tablet_manager.cc:1939] T 490c1633c2c3472b9f28bb974ccb1e6f P 17bdfbef628a4fc3819023753ebe8c35: Deleting consensus metadata
I20260621 07:51:46.268952 13984 ts_tablet_manager.cc:1939] T 490c1633c2c3472b9f28bb974ccb1e6f P eeb16501b843492eadf65978e1008de8: Deleting consensus metadata
I20260621 07:51:46.269526 13438 catalog_manager.cc:4841] TS 17bdfbef628a4fc3819023753ebe8c35 (127.12.12.3:34171): tablet 490c1633c2c3472b9f28bb974ccb1e6f (table test-table [id=ef3aa6713b934f10b2b65de168966efb]) successfully deleted
I20260621 07:51:46.269698 13437 catalog_manager.cc:4841] TS eeb16501b843492eadf65978e1008de8 (127.12.12.1:36481): tablet 490c1633c2c3472b9f28bb974ccb1e6f (table test-table [id=ef3aa6713b934f10b2b65de168966efb]) successfully deleted
I20260621 07:51:46.270104 13983 ts_tablet_manager.cc:1905] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:46.272835 13983 ts_tablet_manager.cc:1918] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:51:46.272919 13983 log.cc:1196] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/490c1633c2c3472b9f28bb974ccb1e6f
I20260621 07:51:46.273160 13983 ts_tablet_manager.cc:1939] T 490c1633c2c3472b9f28bb974ccb1e6f P 7ace1cfa354946b9b9e5196ac0de1136: Deleting consensus metadata
I20260621 07:51:46.273854 13437 catalog_manager.cc:4841] TS 7ace1cfa354946b9b9e5196ac0de1136 (127.12.12.2:40541): tablet 490c1633c2c3472b9f28bb974ccb1e6f (table test-table [id=ef3aa6713b934f10b2b65de168966efb]) successfully deleted
Jun 21 07:51:46 dist-test-slave-lbpt krb5kdc[13406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.12.1@KRBTEST.COM
I20260621 07:51:46.280571 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:46.265074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.12.12.1:36481)
0621 07:51:46.265205 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:46.265210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:46.270836 (+ 5626us) server_negotiation.cc:407] Connection header received
0621 07:51:46.271006 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:46.271009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:46.271063 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:46.271149 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:46.271829 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.272350 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.273311 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.273544 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.273656 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:46.276865 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:46.276878 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:46.276897 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:46.276924 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:46.278850 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:46.279269 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:46.279276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:46.279278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:46.279368 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:46.279606 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:46.279609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:46.279611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:46.279910 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:46.280116 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:46.280271 (+ 155us) server_negotiation.cc:299] Negotiation successful
0621 07:51:46.280392 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260621 07:51:46.281301 13648 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41230
I20260621 07:51:46.291286 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:46.283103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.12.12.62:34251)
0621 07:51:46.283231 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:46.283235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:46.283295 (+ 60us) server_negotiation.cc:407] Connection header received
0621 07:51:46.283394 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:46.283397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:46.283443 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:46.283527 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:46.284120 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.284929 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.285760 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:46.286110 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:46.286207 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:46.287766 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:46.287770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:46.287772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:46.287800 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:46.289393 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:46.289936 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:46.289939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:46.289941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:46.290027 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:46.290270 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:46.290273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:46.290275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:46.290552 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:46.290680 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:46.290992 (+ 312us) server_negotiation.cc:299] Negotiation successful
0621 07:51:46.291112 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260621 07:51:46.291698 13451 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34954
I20260621 07:51:46.293701 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 13523
I20260621 07:51:46.301357 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 13663
I20260621 07:51:46.307206 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 13803
I20260621 07:51:46.312868 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 13419
2026-06-21T07:51:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2635 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:46 dist-test-slave-lbpt krb5kdc[14016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:46 dist-test-slave-lbpt krb5kdc[14016](info): set up 2 sockets
Jun 21 07:51:46 dist-test-slave-lbpt krb5kdc[14016](info): commencing operation
krb5kdc: starting...
W20260621 07:51:47.358385 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:51:47 dist-test-slave-lbpt krb5kdc[14016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:51:47.482024 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:42395
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:42395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:47.595494 14029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:47.595793 14029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:47.595834 14029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:47.599833 14029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:47.599903 14029 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 07:51:47.599921 14029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:47.599939 14029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:47.599956 14029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:47.599972 14029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:47.605417 14029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:42395
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:42395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:47.606575 14029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:47.607577 14029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:51:47.616600 14029 server_base.cc:1034] running on GCE node
W20260621 07:51:47.616490 14034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:47.616531 14037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:47.616496 14035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:47.617400 14029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:47.618629 14029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:47.619808 14029 hybrid_clock.cc:648] HybridClock initialized: now 1782028307619756 us; error 69 us; skew 500 ppm
Jun 21 07:51:47 dist-test-slave-lbpt krb5kdc[14016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028307, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:47.622812 14029 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:51:47.623787 14029 webserver.cc:460] Webserver started at http://127.12.12.62:34793/ using document root <none> and password file <none>
I20260621 07:51:47.624365 14029 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:47.624459 14029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:47.624688 14029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:47.626811 14029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "d8899dd7bde54d85888f5bbb9b467fbb"
format_stamp: "Formatted at 2026-06-21 07:51:47 on dist-test-slave-lbpt"
server_key: "70d84361d2f152cc436c6bd9075a102f"
server_key_iv: "802cc1adb6c1a7500d7527d1a066d182"
server_key_version: "encryptionkey@0"
I20260621 07:51:47.627369 14029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "d8899dd7bde54d85888f5bbb9b467fbb"
format_stamp: "Formatted at 2026-06-21 07:51:47 on dist-test-slave-lbpt"
server_key: "70d84361d2f152cc436c6bd9075a102f"
server_key_iv: "802cc1adb6c1a7500d7527d1a066d182"
server_key_version: "encryptionkey@0"
I20260621 07:51:47.631165 14029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:51:47.633678 14044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:47.635139 14029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:51:47.635274 14029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "d8899dd7bde54d85888f5bbb9b467fbb"
format_stamp: "Formatted at 2026-06-21 07:51:47 on dist-test-slave-lbpt"
server_key: "70d84361d2f152cc436c6bd9075a102f"
server_key_iv: "802cc1adb6c1a7500d7527d1a066d182"
server_key_version: "encryptionkey@0"
I20260621 07:51:47.635385 14029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:47.648772 14029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:47.652194 14029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:47.652405 14029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:47.660816 14029 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:42395
I20260621 07:51:47.660921 14106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:42395 every 8 connection(s)
I20260621 07:51:47.662015 14029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:47.665047 14107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:47.669027 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 14029
I20260621 07:51:47.669127 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:51:47.669335 12336 external_mini_cluster.cc:1428] Setting key 5af2694bf8db78e6694641f32d703a05
I20260621 07:51:47.671926 14107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb: Bootstrap starting.
I20260621 07:51:47.674467 14107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:47.675269 14107 log.cc:824] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:47.677400 14107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb: No bootstrap required, opened a new log
Jun 21 07:51:47 dist-test-slave-lbpt krb5kdc[14016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:47.681401 14107 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8899dd7bde54d85888f5bbb9b467fbb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42395 } }
I20260621 07:51:47.681699 14107 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:47.681769 14107 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8899dd7bde54d85888f5bbb9b467fbb, State: Initialized, Role: FOLLOWER
I20260621 07:51:47.682284 14107 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8899dd7bde54d85888f5bbb9b467fbb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42395 } }
I20260621 07:51:47.682451 14107 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:47.682540 14107 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:47.682693 14107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:47.683802 14107 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8899dd7bde54d85888f5bbb9b467fbb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42395 } }
I20260621 07:51:47.684177 14107 leader_election.cc:302] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8899dd7bde54d85888f5bbb9b467fbb; no voters:
I20260621 07:51:47.684461 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:47.670943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.12.12.62:42395)
0621 07:51:47.671461 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:47.671478 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:47.671518 (+ 40us) server_negotiation.cc:407] Connection header received
0621 07:51:47.672243 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:47.672262 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:47.672735 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:47.673101 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:47.673951 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:47.675056 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:47.675973 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:47.676455 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:47.676630 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:47.679344 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:47.679349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:47.679362 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:47.679389 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:47.681169 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:47.681631 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:47.681637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:47.681641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:47.681790 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:47.682115 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:47.682119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:47.682121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:47.682679 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:47.682982 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:47.683245 (+ 263us) server_negotiation.cc:299] Negotiation successful
0621 07:51:47.683565 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":154,"threads_started":1}
I20260621 07:51:47.684532 14107 leader_election.cc:288] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:47.684604 14112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:47.684810 14112 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [term 1 LEADER]: Becoming Leader. State: Replica: d8899dd7bde54d85888f5bbb9b467fbb, State: Running, Role: LEADER
I20260621 07:51:47.685137 14112 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8899dd7bde54d85888f5bbb9b467fbb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42395 } }
I20260621 07:51:47.685890 14107 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:47.686861 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8899dd7bde54d85888f5bbb9b467fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8899dd7bde54d85888f5bbb9b467fbb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42395 } } }
I20260621 07:51:47.686996 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:47.687297 14114 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8899dd7bde54d85888f5bbb9b467fbb. Latest consensus state: current_term: 1 leader_uuid: "d8899dd7bde54d85888f5bbb9b467fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8899dd7bde54d85888f5bbb9b467fbb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42395 } } }
I20260621 07:51:47.687386 14114 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:47.688622 14121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:47.691521 14121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:47.697346 14121 catalog_manager.cc:1347] Generated new cluster ID: fd66265e11234594a0a9561e78bb59f5
I20260621 07:51:47.697419 14121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:47.710053 14121 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:47.711134 14121 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:47.718307 14121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb: Generated new TSK 0
I20260621 07:51:47.719056 14121 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:51:47.749490 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:42395
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:47.864748 14133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:47.865032 14133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:47.865080 14133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:47.869194 14133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:47.869258 14133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:47.869300 14133 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 07:51:47.869355 14133 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:51:47.869419 14133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:51:47.874949 14133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.62:42395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:47.876034 14133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:47.877156 14133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:47.887462 14141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:47.887491 14139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:47.887490 14138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:47.887737 14133 server_base.cc:1034] running on GCE node
I20260621 07:51:47.888324 14133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:47.888983 14133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:47.890159 14133 hybrid_clock.cc:648] HybridClock initialized: now 1782028307890128 us; error 49 us; skew 500 ppm
Jun 21 07:51:47 dist-test-slave-lbpt krb5kdc[14016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028307, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:47.893095 14133 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:51:47.894033 14133 webserver.cc:460] Webserver started at http://127.12.12.1:42013/ using document root <none> and password file <none>
I20260621 07:51:47.894726 14133 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:47.894824 14133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:47.895056 14133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:47.897176 14133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae"
format_stamp: "Formatted at 2026-06-21 07:51:47 on dist-test-slave-lbpt"
server_key: "51b17f253bd05f821fded9013598b368"
server_key_iv: "e3ea8838a28a1fcbe852547abfeaa8ab"
server_key_version: "encryptionkey@0"
I20260621 07:51:47.897746 14133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae"
format_stamp: "Formatted at 2026-06-21 07:51:47 on dist-test-slave-lbpt"
server_key: "51b17f253bd05f821fded9013598b368"
server_key_iv: "e3ea8838a28a1fcbe852547abfeaa8ab"
server_key_version: "encryptionkey@0"
I20260621 07:51:47.901527 14133 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260621 07:51:47.904247 14148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:47.905671 14133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:51:47.905812 14133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae"
format_stamp: "Formatted at 2026-06-21 07:51:47 on dist-test-slave-lbpt"
server_key: "51b17f253bd05f821fded9013598b368"
server_key_iv: "e3ea8838a28a1fcbe852547abfeaa8ab"
server_key_version: "encryptionkey@0"
I20260621 07:51:47.905933 14133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:47.925946 14133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:47.929663 14133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:47.929867 14133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:47.931488 14133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:47.931562 14133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:47.931623 14133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:47.931699 14133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:51:47 dist-test-slave-lbpt krb5kdc[14016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028307, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:47.943961 14133 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:38061
I20260621 07:51:47.944670 14263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:38061 every 8 connection(s)
I20260621 07:51:47.944981 14133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:51:47.946206 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 14133
I20260621 07:51:47.946347 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:51:47.946748 12336 external_mini_cluster.cc:1428] Setting key 7b9b550f11fa75a835f4f32b1fb29942
I20260621 07:51:47.946954 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:47.932291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:33407 (local address 127.12.12.62:42395)
0621 07:51:47.932450 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:47.932455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:47.933413 (+ 958us) server_negotiation.cc:407] Connection header received
0621 07:51:47.934489 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:47.934493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:47.934550 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:47.934637 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:47.935978 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:47.936575 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:47.938383 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:47.938777 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:47.938901 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:47.942916 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:47.942920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:47.942924 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:47.942955 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:47.944492 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:47.945090 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:47.945094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:47.945096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:47.945240 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:47.945641 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:47.945647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:47.945651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:47.945907 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:47.946054 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:47.946649 (+ 595us) server_negotiation.cc:299] Negotiation successful
0621 07:51:47.946794 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 07:51:47.947847 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:47.932739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:42395 (local address 127.12.12.1:33407)
0621 07:51:47.933236 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:47.933288 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:47.934236 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:47.934829 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:47.934836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:47.935172 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:47.935761 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:47.935783 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:47.936711 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:47.936715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:47.938256 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:47.938269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:47.938955 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:47.938963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:47.939223 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:47.940129 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:47.940167 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:47.942722 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:47.944613 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:47.944621 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:47.944638 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:47.944973 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:47.945334 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:47.945337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:47.945339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:47.945518 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:47.946048 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:47.946054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:47.946333 (+ 279us) client_negotiation.cc:769] Sending connection context
0621 07:51:47.946574 (+ 241us) client_negotiation.cc:240] Negotiation successful
0621 07:51:47.946886 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":161,"threads_started":1}
I20260621 07:51:47.949195 14264 heartbeater.cc:347] Connected to a master server at 127.12.12.62:42395
I20260621 07:51:47.949353 14264 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:47.949934 14264 heartbeater.cc:511] Master 127.12.12.62:42395 requested a full tablet report, sending...
I20260621 07:51:47.951701 14059 ts_manager.cc:194] Registered new tserver with Master: 57ae6638eeaf4b3b974c4f8a5c4b15ae (127.12.12.1:38061)
I20260621 07:51:47.953383 14059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:33407
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
I20260621 07:51:47.962085 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:47.951377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:36913 (local address 127.12.12.62:42395)
0621 07:51:47.951494 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:47.951498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:47.954734 (+ 3236us) server_negotiation.cc:407] Connection header received
0621 07:51:47.954846 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:47.954851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:47.954902 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:47.954995 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:47.955509 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:47.956337 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:47.957154 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:47.957550 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:47.957649 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:47.958890 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:47.958893 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:47.958895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:47.958946 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:47.960440 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:47.960877 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:47.960881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:47.960883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:47.960962 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:47.961247 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:47.961250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:47.961252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:47.961428 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:47.961535 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:47.961803 (+ 268us) server_negotiation.cc:299] Negotiation successful
0621 07:51:47.961917 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:51:47.962090 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:47.951338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:42395 (local address 127.12.12.1:36913)
0621 07:51:47.954620 (+ 3282us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:47.954635 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:47.954754 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:47.955028 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:47.955031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:47.955264 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:47.955375 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:47.955383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:47.956477 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:47.956480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:47.957034 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:47.957042 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:47.957669 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:47.957672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:47.957743 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:47.958286 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:47.958297 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:47.958721 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:47.960557 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:47.960562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:47.960564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:47.960774 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:47.961055 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:47.961059 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:47.961061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:47.961149 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:47.961524 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:47.961528 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:47.961625 (+ 97us) client_negotiation.cc:769] Sending connection context
0621 07:51:47.961738 (+ 113us) client_negotiation.cc:240] Negotiation successful
0621 07:51:47.961892 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3195,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:51:47.975304 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:42395
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:48.094808 14273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:48.095093 14273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:48.095144 14273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:48.099290 14273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:48.099354 14273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:48.099433 14273 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 07:51:48.099468 14273 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:51:48.099516 14273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:51:48.104974 14273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.62:42395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:48.106182 14273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:48.107302 14273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:48.117523 14278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:48.117523 14279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:48.117599 14281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:48.117933 14273 server_base.cc:1034] running on GCE node
I20260621 07:51:48.118455 14273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:48.119154 14273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:48.120349 14273 hybrid_clock.cc:648] HybridClock initialized: now 1782028308120309 us; error 56 us; skew 500 ppm
Jun 21 07:51:48 dist-test-slave-lbpt krb5kdc[14016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028308, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:48.123467 14273 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:51:48.124436 14273 webserver.cc:460] Webserver started at http://127.12.12.2:40131/ using document root <none> and password file <none>
I20260621 07:51:48.125027 14273 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:48.125145 14273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:48.125393 14273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:48.127585 14273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "8807cf54e8534190bd7936951af169e9"
format_stamp: "Formatted at 2026-06-21 07:51:48 on dist-test-slave-lbpt"
server_key: "118be8373c6066e97f741fa7a4629613"
server_key_iv: "42f6a27d895ff3e7610632847248e5a5"
server_key_version: "encryptionkey@0"
I20260621 07:51:48.128185 14273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "8807cf54e8534190bd7936951af169e9"
format_stamp: "Formatted at 2026-06-21 07:51:48 on dist-test-slave-lbpt"
server_key: "118be8373c6066e97f741fa7a4629613"
server_key_iv: "42f6a27d895ff3e7610632847248e5a5"
server_key_version: "encryptionkey@0"
I20260621 07:51:48.132337 14273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:51:48.135218 14288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:48.136525 14273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:51:48.136708 14273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "8807cf54e8534190bd7936951af169e9"
format_stamp: "Formatted at 2026-06-21 07:51:48 on dist-test-slave-lbpt"
server_key: "118be8373c6066e97f741fa7a4629613"
server_key_iv: "42f6a27d895ff3e7610632847248e5a5"
server_key_version: "encryptionkey@0"
I20260621 07:51:48.136839 14273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:48.179783 14273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:48.183276 14273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:48.183491 14273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:48.185030 14273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:48.185110 14273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:48.185181 14273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:48.185228 14273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:51:48 dist-test-slave-lbpt krb5kdc[14016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028308, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:48.196072 14273 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:44483
I20260621 07:51:48.196141 14403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:44483 every 8 connection(s)
I20260621 07:51:48.197286 14273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:51:48.200194 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.185963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:46261 (local address 127.12.12.62:42395)
0621 07:51:48.186158 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:48.186165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:48.187223 (+ 1058us) server_negotiation.cc:407] Connection header received
0621 07:51:48.188273 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:48.188276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:48.188334 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:48.188417 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:48.190297 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.191321 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.192664 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.193058 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.193184 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.196289 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:48.196293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:48.196295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:48.196324 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:48.197967 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:48.198610 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:48.198615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:48.198616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:48.198708 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:48.199058 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:48.199061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:48.199063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:48.199264 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:48.199370 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:48.199882 (+ 512us) server_negotiation.cc:299] Negotiation successful
0621 07:51:48.200010 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 07:51:48.201015 14297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.186513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:42395 (local address 127.12.12.2:46261)
0621 07:51:48.187048 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:48.187100 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:48.188059 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:48.188761 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:48.188772 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:48.189351 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:48.190095 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.190121 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.191471 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.191474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.192550 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.192560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.193248 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.193251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.193425 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.194131 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:48.194156 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:48.196107 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:48.198092 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:48.198101 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:48.198117 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:48.198491 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:48.198814 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:48.198817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:48.198819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:48.198957 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:48.199368 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:48.199375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:48.199627 (+ 252us) client_negotiation.cc:769] Sending connection context
0621 07:51:48.199826 (+ 199us) client_negotiation.cc:240] Negotiation successful
0621 07:51:48.200159 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":145,"threads_started":1}
I20260621 07:51:48.202219 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 14273
I20260621 07:51:48.202337 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:51:48.202457 14404 heartbeater.cc:347] Connected to a master server at 127.12.12.62:42395
I20260621 07:51:48.202682 14404 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:48.202700 12336 external_mini_cluster.cc:1428] Setting key 3ba1c21d164a4cc3555e358d8e48bc39
I20260621 07:51:48.203282 14404 heartbeater.cc:511] Master 127.12.12.62:42395 requested a full tablet report, sending...
I20260621 07:51:48.204437 14061 ts_manager.cc:194] Registered new tserver with Master: 8807cf54e8534190bd7936951af169e9 (127.12.12.2:44483)
I20260621 07:51:48.205304 14061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:46261
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
I20260621 07:51:48.213578 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.204866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:42395 (local address 127.12.12.2:45137)
0621 07:51:48.205146 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:48.205164 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:48.205238 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:48.206338 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:48.206341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:48.206590 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:48.206701 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.206706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.207564 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.207568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.208173 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.208183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.208825 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.208828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.208903 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.209521 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:48.209533 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:48.209854 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:48.211951 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:48.211953 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:48.211955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:48.212147 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:48.212443 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:48.212446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:48.212447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:48.212517 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:48.213032 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:48.213034 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:48.213107 (+ 73us) client_negotiation.cc:769] Sending connection context
0621 07:51:48.213221 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 07:51:48.213373 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":110,"threads_started":1}
I20260621 07:51:48.213594 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.206025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:45137 (local address 127.12.12.62:42395)
0621 07:51:48.206142 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:48.206147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:48.206164 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:51:48.206194 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:48.206196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:48.206230 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:48.206321 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:48.206818 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.207435 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.208300 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.208713 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.208841 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.209999 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:48.210006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:48.210010 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:48.210068 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:48.211835 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:48.212250 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:48.212253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:48.212255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:48.212347 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:48.212607 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:48.212612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:48.212614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:48.212939 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:48.213050 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:48.213275 (+ 225us) server_negotiation.cc:299] Negotiation successful
0621 07:51:48.213408 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:51:48.229352 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:42395
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:48.343524 14413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:48.343808 14413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:48.343858 14413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:48.347963 14413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:48.348029 14413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:48.348073 14413 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 07:51:48.348086 14413 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:51:48.348119 14413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:51:48.353510 14413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.62:42395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:48.354570 14413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:48.355558 14413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:48.365166 14421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:48.365157 14419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:48.365159 14418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:48.365558 14413 server_base.cc:1034] running on GCE node
I20260621 07:51:48.365962 14413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:48.366653 14413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:48.367820 14413 hybrid_clock.cc:648] HybridClock initialized: now 1782028308367805 us; error 29 us; skew 500 ppm
Jun 21 07:51:48 dist-test-slave-lbpt krb5kdc[14016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028308, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:48.370537 14413 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:51:48.371441 14413 webserver.cc:460] Webserver started at http://127.12.12.3:44165/ using document root <none> and password file <none>
I20260621 07:51:48.371968 14413 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:48.372026 14413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:48.372196 14413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:48.374272 14413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "ad96c66d357843d0b6e9618a6f03eb36"
format_stamp: "Formatted at 2026-06-21 07:51:48 on dist-test-slave-lbpt"
server_key: "dbcea6bed005de5b92a0a576a8d6f4f7"
server_key_iv: "5aa0be437d82b124786e6cca7d3e18ab"
server_key_version: "encryptionkey@0"
I20260621 07:51:48.374907 14413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "ad96c66d357843d0b6e9618a6f03eb36"
format_stamp: "Formatted at 2026-06-21 07:51:48 on dist-test-slave-lbpt"
server_key: "dbcea6bed005de5b92a0a576a8d6f4f7"
server_key_iv: "5aa0be437d82b124786e6cca7d3e18ab"
server_key_version: "encryptionkey@0"
I20260621 07:51:48.378793 14413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:51:48.381258 14428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:48.382720 14413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:51:48.382827 14413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "ad96c66d357843d0b6e9618a6f03eb36"
format_stamp: "Formatted at 2026-06-21 07:51:48 on dist-test-slave-lbpt"
server_key: "dbcea6bed005de5b92a0a576a8d6f4f7"
server_key_iv: "5aa0be437d82b124786e6cca7d3e18ab"
server_key_version: "encryptionkey@0"
I20260621 07:51:48.382925 14413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:48.396203 14413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:48.399501 14413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:48.399662 14413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:48.401132 14413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:48.401194 14413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:48.401237 14413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:48.401252 14413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:51:48 dist-test-slave-lbpt krb5kdc[14016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028308, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:48.413069 14413 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:39473
I20260621 07:51:48.413086 14543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:39473 every 8 connection(s)
I20260621 07:51:48.414207 14413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:51:48.415814 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.402161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:42047 (local address 127.12.12.62:42395)
0621 07:51:48.402298 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:48.402301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:48.403475 (+ 1174us) server_negotiation.cc:407] Connection header received
0621 07:51:48.404764 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:48.404768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:48.404830 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:48.404937 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:48.406292 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.406893 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.408176 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.408403 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.408524 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.411772 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:48.411777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:48.411785 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:48.411813 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:48.413422 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:48.414047 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:48.414050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:48.414051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:48.414126 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:48.414522 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:48.414526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:48.414527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:48.414775 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:48.414904 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:48.415483 (+ 579us) server_negotiation.cc:299] Negotiation successful
0621 07:51:48.415620 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:51:48.415984 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 14413
I20260621 07:51:48.416100 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:51:48.416385 12336 external_mini_cluster.cc:1428] Setting key f1e48c94fa2ff471b88a8f5c82fcdedd
I20260621 07:51:48.416738 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.402770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:42395 (local address 127.12.12.3:42047)
0621 07:51:48.403303 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:48.403356 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:48.404542 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:48.405108 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:48.405116 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:48.405538 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:48.406142 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.406157 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.407026 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.407034 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.408037 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.408047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.408576 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.408580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.408774 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.409485 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:48.409514 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:48.411582 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:48.413559 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:48.413565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:48.413579 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:48.413942 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:48.414230 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:48.414234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:48.414236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:48.414405 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:48.414899 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:48.414905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:48.415209 (+ 304us) client_negotiation.cc:769] Sending connection context
0621 07:51:48.415427 (+ 218us) client_negotiation.cc:240] Negotiation successful
0621 07:51:48.415763 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":145,"threads_started":1}
I20260621 07:51:48.418481 14544 heartbeater.cc:347] Connected to a master server at 127.12.12.62:42395
I20260621 07:51:48.418622 14544 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:48.419203 14544 heartbeater.cc:511] Master 127.12.12.62:42395 requested a full tablet report, sending...
I20260621 07:51:48.420248 14059 ts_manager.cc:194] Registered new tserver with Master: ad96c66d357843d0b6e9618a6f03eb36 (127.12.12.3:39473)
I20260621 07:51:48.421077 14059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:42047
I20260621 07:51:48.427282 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.420569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:51337 (local address 127.12.12.62:42395)
0621 07:51:48.420670 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:48.420674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:48.420873 (+ 199us) server_negotiation.cc:407] Connection header received
0621 07:51:48.420951 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:48.420954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:48.420992 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:48.421062 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:48.421549 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.422302 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.423169 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.423374 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.423455 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.424516 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:48.424519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:48.424521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:48.424542 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:48.425811 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:48.426185 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:48.426187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:48.426189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:48.426265 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:48.426536 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:48.426538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:48.426539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:48.426740 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:48.426820 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:48.427078 (+ 258us) server_negotiation.cc:299] Negotiation successful
0621 07:51:48.427167 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260621 07:51:48.427284 14550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.420451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:42395 (local address 127.12.12.3:51337)
0621 07:51:48.420755 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:48.420770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:48.420870 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:48.421105 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:48.421109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:48.421322 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:48.421455 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.421463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.422479 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.422483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.423049 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.423057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.423475 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.423477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.423548 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.424049 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:48.424060 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:48.424392 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:48.425910 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:48.425912 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:48.425914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:48.426099 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:48.426344 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:48.426346 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:48.426347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:48.426469 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:48.426848 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:48.426851 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:48.426928 (+ 77us) client_negotiation.cc:769] Sending connection context
0621 07:51:48.427024 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 07:51:48.427132 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":88,"threads_started":1}
I20260621 07:51:48.430496 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 07:51:48 dist-test-slave-lbpt krb5kdc[14016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 07:51:48 dist-test-slave-lbpt krb5kdc[14016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:48.456355 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.447754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.12.12.62:42395)
0621 07:51:48.447900 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:48.447904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:48.447916 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:51:48.447956 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:48.447958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:48.447997 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:48.448089 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:48.448813 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.449462 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.450346 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.450635 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.450749 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.453355 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:48.453358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:48.453360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:48.453388 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:48.454698 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:48.455114 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:48.455119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:48.455121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:48.455210 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:48.455483 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:48.455488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:48.455490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:48.455682 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:48.455823 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:48.456061 (+ 238us) server_negotiation.cc:299] Negotiation successful
0621 07:51:48.456189 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 07:51:48.459683 14059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:51:48.462935 14059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:51:48.477566 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.471832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:38061 (local address 127.0.0.1:44328)
0621 07:51:48.472218 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:48.472278 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:48.472407 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:48.473228 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:48.473236 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:48.473266 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:48.473420 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.473427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.474813 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.474816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.476097 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.476106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.477018 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.477022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.477130 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.477168 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 07:51:48.477283 (+ 115us) client_negotiation.cc:240] Negotiation successful
0621 07:51:48.477412 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":102,"threads_started":1}
I20260621 07:51:48.477566 14562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.472030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.12.12.1:38061)
0621 07:51:48.472401 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:48.472408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:48.472435 (+ 27us) server_negotiation.cc:407] Connection header received
0621 07:51:48.472635 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:48.472641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:48.472868 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:48.473018 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:48.473548 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.474693 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.476228 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.476885 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.477014 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.477188 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:48.477312 (+ 124us) server_negotiation.cc:299] Negotiation successful
0621 07:51:48.477412 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":75,"threads_started":1}
I20260621 07:51:48.478658 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.472600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:44483 (local address 127.0.0.1:38968)
0621 07:51:48.473216 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:48.473229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:48.473301 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:48.473957 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:48.473960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:48.473977 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:48.474098 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.474112 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.475444 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.475449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.476999 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.477011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.478104 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.478107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.478302 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.478326 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 07:51:48.478403 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 07:51:48.478503 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":230,"thread_start_us":105,"threads_started":1}
I20260621 07:51:48.478663 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.472101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:39473 (local address 127.0.0.1:41748)
0621 07:51:48.472736 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:48.472750 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:48.472842 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:48.474024 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:48.474027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:48.474047 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:48.474147 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.474152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.475387 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.475392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.476999 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.477011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.478132 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.478136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.478257 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.478302 (+ 45us) client_negotiation.cc:769] Sending connection context
0621 07:51:48.478359 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 07:51:48.478479 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":209,"thread_start_us":122,"threads_started":1}
I20260621 07:51:48.478757 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.473043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.12.12.2:44483)
0621 07:51:48.473521 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:48.473536 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:48.473561 (+ 25us) server_negotiation.cc:407] Connection header received
0621 07:51:48.473638 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:48.473642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:48.473818 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:48.473972 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:48.474219 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.475349 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.477150 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.477977 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.478106 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.478265 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:48.478497 (+ 232us) server_negotiation.cc:299] Negotiation successful
0621 07:51:48.478608 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":133,"threads_started":1}
I20260621 07:51:48.479447 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.472684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.12.12.3:39473)
0621 07:51:48.473564 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:48.473577 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:48.473600 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:51:48.473674 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:48.473679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:48.473868 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:48.474290 (+ 422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:48.474439 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.475266 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.477149 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.477976 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.478959 (+ 983us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.479145 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:48.479198 (+ 53us) server_negotiation.cc:299] Negotiation successful
0621 07:51:48.479297 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":108,"threads_started":1}
I20260621 07:51:48.481144 14198 tablet_service.cc:1467] Processing CreateTablet for tablet 10e7fc01cda74606a55c2da9a3c9a345 (DEFAULT_TABLE table=test-table [id=dbc7dc57331144dcaf49757f7e3ac153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:48.482153 14198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e7fc01cda74606a55c2da9a3c9a345. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:48.481861 14338 tablet_service.cc:1467] Processing CreateTablet for tablet 10e7fc01cda74606a55c2da9a3c9a345 (DEFAULT_TABLE table=test-table [id=dbc7dc57331144dcaf49757f7e3ac153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:48.481861 14478 tablet_service.cc:1467] Processing CreateTablet for tablet 10e7fc01cda74606a55c2da9a3c9a345 (DEFAULT_TABLE table=test-table [id=dbc7dc57331144dcaf49757f7e3ac153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:48.482932 14338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e7fc01cda74606a55c2da9a3c9a345. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:48.482931 14478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e7fc01cda74606a55c2da9a3c9a345. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:48.489230 14567 tablet_bootstrap.cc:492] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Bootstrap starting.
I20260621 07:51:48.491786 14568 tablet_bootstrap.cc:492] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9: Bootstrap starting.
I20260621 07:51:48.492333 14567 tablet_bootstrap.cc:654] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:48.493338 14567 log.cc:824] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:48.493651 14569 tablet_bootstrap.cc:492] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36: Bootstrap starting.
I20260621 07:51:48.494095 14568 tablet_bootstrap.cc:654] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:48.494976 14568 log.cc:824] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:48.495775 14569 tablet_bootstrap.cc:654] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:48.495949 14567 tablet_bootstrap.cc:492] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: No bootstrap required, opened a new log
I20260621 07:51:48.496150 14567 ts_tablet_manager.cc:1397] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260621 07:51:48.496644 14569 log.cc:824] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:48.497119 14568 tablet_bootstrap.cc:492] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9: No bootstrap required, opened a new log
I20260621 07:51:48.497305 14568 ts_tablet_manager.cc:1397] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:51:48.498477 14569 tablet_bootstrap.cc:492] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36: No bootstrap required, opened a new log
I20260621 07:51:48.498665 14569 ts_tablet_manager.cc:1397] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:51:48.501138 14567 raft_consensus.cc:348] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:48.501245 14568 raft_consensus.cc:348] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:48.501451 14568 raft_consensus.cc:374] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:48.501451 14567 raft_consensus.cc:374] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:48.501499 14568 raft_consensus.cc:729] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8807cf54e8534190bd7936951af169e9, State: Initialized, Role: FOLLOWER
I20260621 07:51:48.501499 14567 raft_consensus.cc:729] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ae6638eeaf4b3b974c4f8a5c4b15ae, State: Initialized, Role: FOLLOWER
I20260621 07:51:48.501754 14569 raft_consensus.cc:348] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:48.502038 14569 raft_consensus.cc:374] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:48.502100 14569 raft_consensus.cc:729] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad96c66d357843d0b6e9618a6f03eb36, State: Initialized, Role: FOLLOWER
I20260621 07:51:48.502030 14567 consensus_queue.cc:260] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:48.502058 14568 consensus_queue.cc:260] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:48.502562 14569 consensus_queue.cc:260] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:48.503150 14567 ts_tablet_manager.cc:1428] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260621 07:51:48.503314 14544 heartbeater.cc:503] Master 127.12.12.62:42395 was elected leader, sending a full tablet report...
I20260621 07:51:48.503545 14264 heartbeater.cc:503] Master 127.12.12.62:42395 was elected leader, sending a full tablet report...
I20260621 07:51:48.503599 14569 ts_tablet_manager.cc:1428] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:51:48.503767 14568 ts_tablet_manager.cc:1428] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260621 07:51:48.503887 14404 heartbeater.cc:503] Master 127.12.12.62:42395 was elected leader, sending a full tablet report...
W20260621 07:51:48.665485 14545 tablet.cc:2369] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:51:48.696439 14265 tablet.cc:2369] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:51:48.698606 14405 tablet.cc:2369] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:51:48.706281 14574 raft_consensus.cc:482] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:51:48.706555 14574 raft_consensus.cc:504] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:48.707770 14574 leader_election.cc:288] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57ae6638eeaf4b3b974c4f8a5c4b15ae (127.12.12.1:38061), ad96c66d357843d0b6e9618a6f03eb36 (127.12.12.3:39473)
I20260621 07:51:48.712046 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.708156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:36121 (local address 127.12.12.3:39473)
0621 07:51:48.708290 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:48.708294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:48.708555 (+ 261us) server_negotiation.cc:407] Connection header received
0621 07:51:48.708739 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:48.708742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:48.708794 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:48.708846 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:48.709232 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.709899 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.711049 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.711467 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.711551 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.711646 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:48.711825 (+ 179us) server_negotiation.cc:299] Negotiation successful
0621 07:51:48.711906 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:51:48.712072 14562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.708129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:51115 (local address 127.12.12.1:38061)
0621 07:51:48.708281 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:48.708286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:48.708471 (+ 185us) server_negotiation.cc:407] Connection header received
0621 07:51:48.708669 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:48.708672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:48.708742 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:48.708839 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:48.709119 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.709815 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.711096 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.711503 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.711635 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.711693 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:48.711885 (+ 192us) server_negotiation.cc:299] Negotiation successful
0621 07:51:48.711965 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 07:51:48.712090 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.708050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:39473 (local address 127.12.12.2:36121)
0621 07:51:48.708456 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:48.708471 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:48.708585 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:48.708978 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:48.708979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:48.708993 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:48.709118 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.709124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.710013 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.710016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.710932 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.710942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.711588 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.711595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.711690 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.711716 (+ 26us) client_negotiation.cc:769] Sending connection context
0621 07:51:48.711790 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 07:51:48.711883 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":22,"thread_start_us":78,"threads_started":1}
I20260621 07:51:48.712306 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.708004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:38061 (local address 127.12.12.2:51115)
0621 07:51:48.708365 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:48.708377 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:48.708516 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:48.708849 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:48.708852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:48.708866 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:48.708992 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.708999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.709925 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.709931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.710942 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.710951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.711646 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.711650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.711746 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.711762 (+ 16us) client_negotiation.cc:769] Sending connection context
0621 07:51:48.711857 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 07:51:48.711924 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":27,"thread_start_us":109,"threads_started":1}
I20260621 07:51:48.712855 14498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e7fc01cda74606a55c2da9a3c9a345" candidate_uuid: "8807cf54e8534190bd7936951af169e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad96c66d357843d0b6e9618a6f03eb36" is_pre_election: true
I20260621 07:51:48.712917 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e7fc01cda74606a55c2da9a3c9a345" candidate_uuid: "8807cf54e8534190bd7936951af169e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" is_pre_election: true
I20260621 07:51:48.713192 14218 raft_consensus.cc:2454] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8807cf54e8534190bd7936951af169e9 in term 0.
I20260621 07:51:48.713192 14498 raft_consensus.cc:2454] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8807cf54e8534190bd7936951af169e9 in term 0.
I20260621 07:51:48.713717 14291 leader_election.cc:302] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 57ae6638eeaf4b3b974c4f8a5c4b15ae, 8807cf54e8534190bd7936951af169e9, ad96c66d357843d0b6e9618a6f03eb36; no voters:
I20260621 07:51:48.713989 14574 raft_consensus.cc:2781] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:51:48.714087 14574 raft_consensus.cc:482] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:51:48.714161 14574 raft_consensus.cc:3037] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:48.715227 14574 raft_consensus.cc:504] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:48.715569 14574 leader_election.cc:288] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [CANDIDATE]: Term 1 election: Requested vote from peers 57ae6638eeaf4b3b974c4f8a5c4b15ae (127.12.12.1:38061), ad96c66d357843d0b6e9618a6f03eb36 (127.12.12.3:39473)
I20260621 07:51:48.715956 14498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e7fc01cda74606a55c2da9a3c9a345" candidate_uuid: "8807cf54e8534190bd7936951af169e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad96c66d357843d0b6e9618a6f03eb36"
I20260621 07:51:48.715956 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e7fc01cda74606a55c2da9a3c9a345" candidate_uuid: "8807cf54e8534190bd7936951af169e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae"
I20260621 07:51:48.716084 14218 raft_consensus.cc:3037] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:48.716084 14498 raft_consensus.cc:3037] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:48.717486 14218 raft_consensus.cc:2454] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8807cf54e8534190bd7936951af169e9 in term 1.
I20260621 07:51:48.717675 14498 raft_consensus.cc:2454] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8807cf54e8534190bd7936951af169e9 in term 1.
I20260621 07:51:48.717902 14289 leader_election.cc:302] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ae6638eeaf4b3b974c4f8a5c4b15ae, 8807cf54e8534190bd7936951af169e9; no voters:
I20260621 07:51:48.718101 14574 raft_consensus.cc:2781] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:48.718295 14574 raft_consensus.cc:686] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [term 1 LEADER]: Becoming Leader. State: Replica: 8807cf54e8534190bd7936951af169e9, State: Running, Role: LEADER
I20260621 07:51:48.718659 14574 consensus_queue.cc:237] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:48.721927 14061 catalog_manager.cc:5495] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8807cf54e8534190bd7936951af169e9 (127.12.12.2). New cstate: current_term: 1 leader_uuid: "8807cf54e8534190bd7936951af169e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } health_report { overall_health: UNKNOWN } } }
Jun 21 07:51:48 dist-test-slave-lbpt krb5kdc[14016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028307, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:48.793149 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.784217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:42395 (local address 127.0.0.1:46766)
0621 07:51:48.784514 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:48.784527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:48.784675 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:48.784903 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:48.784905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:48.785303 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:48.785435 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.785449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.786523 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.786527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.787360 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:48.787369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.787869 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.787873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:48.787969 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.788549 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:48.788561 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:48.790037 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:48.791753 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:48.791761 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:48.791767 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:48.792011 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:48.792253 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:48.792255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:48.792256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:48.792336 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:48.792662 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:48.792668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:48.792771 (+ 103us) client_negotiation.cc:769] Sending connection context
0621 07:51:48.792869 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 07:51:48.792993 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":101,"threads_started":1}
I20260621 07:51:48.793160 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:48.784323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.12.12.62:42395)
0621 07:51:48.784494 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:48.784499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:48.784636 (+ 137us) server_negotiation.cc:407] Connection header received
0621 07:51:48.784755 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:48.784758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:48.784797 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:48.784869 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:48.785576 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.786330 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.787490 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:48.787742 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:48.787870 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:48.790175 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:48.790178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:48.790180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:48.790203 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:48.791649 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:48.792099 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:48.792102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:48.792103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:48.792172 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:48.792410 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:48.792412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:48.792413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:48.792571 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:48.792644 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:48.792923 (+ 279us) server_negotiation.cc:299] Negotiation successful
0621 07:51:48.793048 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":90}
I20260621 07:51:48.797288 14061 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:46766:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 07:51:48.798121 14061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:51:48.803316 14338 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbae6d204e4491db24e5ef245a2d581 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a202596262a04ae18cc32953b71f7259]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:51:48.803133 14198 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbae6d204e4491db24e5ef245a2d581 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a202596262a04ae18cc32953b71f7259]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:51:48.803440 14478 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbae6d204e4491db24e5ef245a2d581 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a202596262a04ae18cc32953b71f7259]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:51:48.803928 14338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbae6d204e4491db24e5ef245a2d581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:48.803928 14198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbae6d204e4491db24e5ef245a2d581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:48.804162 14478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbae6d204e4491db24e5ef245a2d581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:48.806797 14569 tablet_bootstrap.cc:492] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36: Bootstrap starting.
I20260621 07:51:48.807112 14568 tablet_bootstrap.cc:492] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9: Bootstrap starting.
I20260621 07:51:48.807734 14569 tablet_bootstrap.cc:654] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:48.808248 14568 tablet_bootstrap.cc:654] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:48.808449 14567 tablet_bootstrap.cc:492] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Bootstrap starting.
I20260621 07:51:48.809142 14569 tablet_bootstrap.cc:492] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36: No bootstrap required, opened a new log
I20260621 07:51:48.809206 14569 ts_tablet_manager.cc:1397] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:48.809486 14567 tablet_bootstrap.cc:654] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:48.809583 14569 raft_consensus.cc:348] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:48.809717 14569 raft_consensus.cc:374] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:48.809744 14569 raft_consensus.cc:729] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad96c66d357843d0b6e9618a6f03eb36, State: Initialized, Role: FOLLOWER
I20260621 07:51:48.809844 14569 consensus_queue.cc:260] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:48.810086 14569 ts_tablet_manager.cc:1428] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260621 07:51:48.810120 14575 tablet_replica.cc:439] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } } }
I20260621 07:51:48.810266 14575 tablet_replica.cc:442] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:51:48.811031 14567 tablet_bootstrap.cc:492] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: No bootstrap required, opened a new log
I20260621 07:51:48.811059 14568 tablet_bootstrap.cc:492] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9: No bootstrap required, opened a new log
I20260621 07:51:48.811142 14567 ts_tablet_manager.cc:1397] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 07:51:48.811164 14568 ts_tablet_manager.cc:1397] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260621 07:51:48.811568 14568 raft_consensus.cc:348] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:48.811681 14568 raft_consensus.cc:374] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:48.811722 14568 raft_consensus.cc:729] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8807cf54e8534190bd7936951af169e9, State: Initialized, Role: FOLLOWER
I20260621 07:51:48.811672 14567 raft_consensus.cc:348] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:48.811828 14567 raft_consensus.cc:374] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:48.811885 14567 raft_consensus.cc:729] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ae6638eeaf4b3b974c4f8a5c4b15ae, State: Initialized, Role: FOLLOWER
I20260621 07:51:48.811862 14568 consensus_queue.cc:260] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:48.812012 14567 consensus_queue.cc:260] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:48.812112 14574 tablet_replica.cc:439] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } } }
I20260621 07:51:48.812227 14574 tablet_replica.cc:442] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:51:48.812290 14567 ts_tablet_manager.cc:1428] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260621 07:51:48.812434 14568 ts_tablet_manager.cc:1428] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:51:48.812651 14573 tablet_replica.cc:439] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } } }
I20260621 07:51:48.812799 14573 tablet_replica.cc:442] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:51:49.052948 14574 raft_consensus.cc:482] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:51:49.053121 14574 raft_consensus.cc:504] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:49.053597 14574 leader_election.cc:288] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57ae6638eeaf4b3b974c4f8a5c4b15ae (127.12.12.1:38061), ad96c66d357843d0b6e9618a6f03eb36 (127.12.12.3:39473)
I20260621 07:51:49.054111 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbae6d204e4491db24e5ef245a2d581" candidate_uuid: "8807cf54e8534190bd7936951af169e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" is_pre_election: true
I20260621 07:51:49.054298 14218 raft_consensus.cc:2454] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8807cf54e8534190bd7936951af169e9 in term 0.
I20260621 07:51:49.054250 14498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbae6d204e4491db24e5ef245a2d581" candidate_uuid: "8807cf54e8534190bd7936951af169e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad96c66d357843d0b6e9618a6f03eb36" is_pre_election: true
I20260621 07:51:49.054507 14498 raft_consensus.cc:2454] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8807cf54e8534190bd7936951af169e9 in term 0.
I20260621 07:51:49.054733 14289 leader_election.cc:302] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ae6638eeaf4b3b974c4f8a5c4b15ae, 8807cf54e8534190bd7936951af169e9; no voters:
I20260621 07:51:49.054936 14574 raft_consensus.cc:2781] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:51:49.055007 14574 raft_consensus.cc:482] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:51:49.055050 14574 raft_consensus.cc:3037] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:49.056072 14574 raft_consensus.cc:504] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:49.056411 14574 leader_election.cc:288] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [CANDIDATE]: Term 1 election: Requested vote from peers 57ae6638eeaf4b3b974c4f8a5c4b15ae (127.12.12.1:38061), ad96c66d357843d0b6e9618a6f03eb36 (127.12.12.3:39473)
I20260621 07:51:49.056828 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbae6d204e4491db24e5ef245a2d581" candidate_uuid: "8807cf54e8534190bd7936951af169e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae"
I20260621 07:51:49.056828 14498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbae6d204e4491db24e5ef245a2d581" candidate_uuid: "8807cf54e8534190bd7936951af169e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad96c66d357843d0b6e9618a6f03eb36"
I20260621 07:51:49.056968 14498 raft_consensus.cc:3037] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:49.056968 14218 raft_consensus.cc:3037] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:49.058121 14498 raft_consensus.cc:2454] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8807cf54e8534190bd7936951af169e9 in term 1.
I20260621 07:51:49.058136 14218 raft_consensus.cc:2454] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8807cf54e8534190bd7936951af169e9 in term 1.
I20260621 07:51:49.058480 14289 leader_election.cc:302] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 57ae6638eeaf4b3b974c4f8a5c4b15ae, 8807cf54e8534190bd7936951af169e9, ad96c66d357843d0b6e9618a6f03eb36; no voters:
I20260621 07:51:49.058727 14574 raft_consensus.cc:2781] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:49.058830 14574 raft_consensus.cc:686] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [term 1 LEADER]: Becoming Leader. State: Replica: 8807cf54e8534190bd7936951af169e9, State: Running, Role: LEADER
I20260621 07:51:49.058976 14574 consensus_queue.cc:237] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:49.059633 14578 tablet_replica.cc:439] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9: TxnStatusTablet state changed. Reason: New leader 8807cf54e8534190bd7936951af169e9. Latest consensus state: current_term: 1 leader_uuid: "8807cf54e8534190bd7936951af169e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } } }
I20260621 07:51:49.059746 14578 tablet_replica.cc:442] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:51:49.060010 14590 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 07:51:49.060463 14590 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 07:51:49.061010 14061 catalog_manager.cc:5495] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8807cf54e8534190bd7936951af169e9 (127.12.12.2). New cstate: current_term: 1 leader_uuid: "8807cf54e8534190bd7936951af169e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } health_report { overall_health: UNKNOWN } } }
I20260621 07:51:49.114893 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:49.110523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:44483 (local address 127.0.0.1:38980)
0621 07:51:49.110757 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:49.110774 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:49.110889 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:49.111212 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:49.111215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:49.111224 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:51:49.111336 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:49.111342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.112149 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.112153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:49.113143 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:49.113155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.113623 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.113627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:49.113728 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:49.113789 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:51:49.114543 (+ 754us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:51:49.114560 (+ 17us) client_negotiation.cc:769] Sending connection context
0621 07:51:49.114636 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 07:51:49.114735 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260621 07:51:49.114935 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:49.110614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38980 (local address 127.12.12.2:44483)
0621 07:51:49.110903 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:49.110911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:49.110925 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:51:49.111018 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:49.111020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:49.111074 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:49.111172 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:49.111449 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.112026 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.113267 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.113506 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.113584 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:49.113937 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:49.114451 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:49.114537 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:49.114656 (+ 119us) server_negotiation.cc:299] Negotiation successful
0621 07:51:49.114757 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":116,"threads_started":1}
I20260621 07:51:49.120491 14498 raft_consensus.cc:1261] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Refusing update from remote peer 8807cf54e8534190bd7936951af169e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:49.120523 14218 raft_consensus.cc:1261] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 FOLLOWER]: Refusing update from remote peer 8807cf54e8534190bd7936951af169e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:49.121194 14578 consensus_queue.cc:1035] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:49.121404 14574 consensus_queue.cc:1035] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:49.124902 14575 tablet_replica.cc:439] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36: TxnStatusTablet state changed. Reason: New leader 8807cf54e8534190bd7936951af169e9. Latest consensus state: current_term: 1 leader_uuid: "8807cf54e8534190bd7936951af169e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } } }
I20260621 07:51:49.125020 14575 tablet_replica.cc:442] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:51:49.125349 14573 tablet_replica.cc:439] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: TxnStatusTablet state changed. Reason: New leader 8807cf54e8534190bd7936951af169e9. Latest consensus state: current_term: 1 leader_uuid: "8807cf54e8534190bd7936951af169e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } } }
I20260621 07:51:49.125463 14573 tablet_replica.cc:442] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:51:49.128221 14578 tablet_replica.cc:439] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8807cf54e8534190bd7936951af169e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } } }
I20260621 07:51:49.128331 14578 tablet_replica.cc:442] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:51:49.128482 14575 tablet_replica.cc:439] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8807cf54e8534190bd7936951af169e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } } }
I20260621 07:51:49.128602 14575 tablet_replica.cc:442] T 7cbae6d204e4491db24e5ef245a2d581 P ad96c66d357843d0b6e9618a6f03eb36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:51:49.128626 14596 mvcc.cc:205] Tried to move back new op lower bound from 7299187954150125568 to 7299187953906745344. Current Snapshot: MvccSnapshot[applied={T|T < 7299187954150125568}]
I20260621 07:51:49.130486 14578 tablet_replica.cc:439] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8807cf54e8534190bd7936951af169e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } } }
I20260621 07:51:49.130558 14594 mvcc.cc:205] Tried to move back new op lower bound from 7299187954150125568 to 7299187953906745344. Current Snapshot: MvccSnapshot[applied={T|T < 7299187954150125568}]
I20260621 07:51:49.130604 14578 tablet_replica.cc:442] T 7cbae6d204e4491db24e5ef245a2d581 P 8807cf54e8534190bd7936951af169e9: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:51:49.131680 14573 tablet_replica.cc:439] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8807cf54e8534190bd7936951af169e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } } }
I20260621 07:51:49.131798 14573 tablet_replica.cc:442] T 7cbae6d204e4491db24e5ef245a2d581 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:51:49.132937 14595 mvcc.cc:205] Tried to move back new op lower bound from 7299187954150125568 to 7299187953906745344. Current Snapshot: MvccSnapshot[applied={T|T < 7299187954150125568}]
I20260621 07:51:49.140173 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:49.136360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38990 (local address 127.12.12.2:44483)
0621 07:51:49.136485 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:49.136490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:49.136670 (+ 180us) server_negotiation.cc:407] Connection header received
0621 07:51:49.136823 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:49.136827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:49.136875 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:49.136955 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:49.137177 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.137952 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.138871 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.139248 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.139347 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:49.139703 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:49.139776 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:49.139845 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:49.139960 (+ 115us) server_negotiation.cc:299] Negotiation successful
0621 07:51:49.140007 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:51:49.151556 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:49.147853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:58419 (local address 127.12.12.2:44483)
0621 07:51:49.147966 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:49.147970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:49.148219 (+ 249us) server_negotiation.cc:407] Connection header received
0621 07:51:49.148334 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:49.148344 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:49.148384 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:49.148445 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:49.148754 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.149252 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.150525 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.150737 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.150817 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:49.151140 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:49.151207 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:49.151260 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:49.151376 (+ 116us) server_negotiation.cc:299] Negotiation successful
0621 07:51:49.151418 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":47}
I20260621 07:51:49.151623 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:49.147798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:44483 (local address 127.12.12.2:58419)
0621 07:51:49.148128 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:49.148143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:49.148247 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:49.148476 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:49.148479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:49.148487 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:51:49.148614 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:49.148619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.149344 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.149347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:49.150357 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:49.150368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.150845 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.150848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:49.150934 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:49.150985 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:51:49.151286 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:51:49.151292 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:51:49.151358 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 07:51:49.151437 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":18,"thread_start_us":101,"threads_started":1}
I20260621 07:51:49.158294 14218 raft_consensus.cc:1261] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 FOLLOWER]: Refusing update from remote peer 8807cf54e8534190bd7936951af169e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:49.158375 14498 raft_consensus.cc:1261] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Refusing update from remote peer 8807cf54e8534190bd7936951af169e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:49.158723 14578 consensus_queue.cc:1035] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:49.158890 14574 consensus_queue.cc:1035] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:49.203244 14061 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:49.203454 14061 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:49.206454 14061 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb: Sending DeleteTablet for 3 replicas of tablet 10e7fc01cda74606a55c2da9a3c9a345
I20260621 07:51:49.207250 14338 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e7fc01cda74606a55c2da9a3c9a345 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:49 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:38968
I20260621 07:51:49.207368 14198 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e7fc01cda74606a55c2da9a3c9a345 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:49 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:44328
I20260621 07:51:49.207901 14613 tablet_replica.cc:330] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: stopping tablet replica
I20260621 07:51:49.208225 14613 raft_consensus.cc:2229] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:49.208428 14613 raft_consensus.cc:2258] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:49.208595 14478 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e7fc01cda74606a55c2da9a3c9a345 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:49 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:41748
I20260621 07:51:49.209810 14613 ts_tablet_manager.cc:1905] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:49.209937 14061 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:51:49.210392 14061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:51:49.213279 14613 ts_tablet_manager.cc:1918] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:51:49.213377 14613 log.cc:1196] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/wal/wals/10e7fc01cda74606a55c2da9a3c9a345
I20260621 07:51:49.213819 14613 ts_tablet_manager.cc:1939] T 10e7fc01cda74606a55c2da9a3c9a345 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Deleting consensus metadata
I20260621 07:51:49.214694 14614 tablet_replica.cc:330] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36: stopping tablet replica
I20260621 07:51:49.215118 14614 raft_consensus.cc:2229] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:49.215344 14045 catalog_manager.cc:4841] TS 57ae6638eeaf4b3b974c4f8a5c4b15ae (127.12.12.1:38061): tablet 10e7fc01cda74606a55c2da9a3c9a345 (table test-table [id=dbc7dc57331144dcaf49757f7e3ac153]) successfully deleted
I20260621 07:51:49.215409 14614 raft_consensus.cc:2258] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:49.216369 14478 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa198344cc04e6a9d8d7301e1932573 (DEFAULT_TABLE table=test-table [id=9ab225923cf94938bb90a3113a0d8ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:49.216643 14478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa198344cc04e6a9d8d7301e1932573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:49.216614 14338 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa198344cc04e6a9d8d7301e1932573 (DEFAULT_TABLE table=test-table [id=9ab225923cf94938bb90a3113a0d8ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:49.216853 14338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa198344cc04e6a9d8d7301e1932573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:49.218971 14569 tablet_bootstrap.cc:492] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36: Bootstrap starting.
I20260621 07:51:49.219246 14568 tablet_bootstrap.cc:492] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9: Bootstrap starting.
I20260621 07:51:49.219349 14198 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa198344cc04e6a9d8d7301e1932573 (DEFAULT_TABLE table=test-table [id=9ab225923cf94938bb90a3113a0d8ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:49.219574 14198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa198344cc04e6a9d8d7301e1932573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:49.219900 14569 tablet_bootstrap.cc:654] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:49.219969 14568 tablet_bootstrap.cc:654] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:49.221434 14614 ts_tablet_manager.cc:1905] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:49.221938 14567 tablet_bootstrap.cc:492] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Bootstrap starting.
I20260621 07:51:49.222198 14569 tablet_bootstrap.cc:492] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36: No bootstrap required, opened a new log
I20260621 07:51:49.222285 14569 ts_tablet_manager.cc:1397] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 07:51:49.222754 14569 raft_consensus.cc:348] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:49.222883 14569 raft_consensus.cc:374] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:49.222923 14569 raft_consensus.cc:729] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad96c66d357843d0b6e9618a6f03eb36, State: Initialized, Role: FOLLOWER
I20260621 07:51:49.223071 14569 consensus_queue.cc:260] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:49.223398 14569 ts_tablet_manager.cc:1428] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260621 07:51:49.223419 14567 tablet_bootstrap.cc:654] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:49.223568 14575 raft_consensus.cc:482] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:51:49.223663 14575 raft_consensus.cc:504] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:49.224042 14612 tablet_replica.cc:330] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9: stopping tablet replica
I20260621 07:51:49.224359 14612 raft_consensus.cc:2229] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:51:49.224890 14575 leader_election.cc:288] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57ae6638eeaf4b3b974c4f8a5c4b15ae (127.12.12.1:38061), 8807cf54e8534190bd7936951af169e9 (127.12.12.2:44483)
I20260621 07:51:49.225102 14614 ts_tablet_manager.cc:1918] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:51:49.225251 14614 log.cc:1196] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/10e7fc01cda74606a55c2da9a3c9a345
I20260621 07:51:49.225633 14614 ts_tablet_manager.cc:1939] T 10e7fc01cda74606a55c2da9a3c9a345 P ad96c66d357843d0b6e9618a6f03eb36: Deleting consensus metadata
I20260621 07:51:49.226986 14047 catalog_manager.cc:4841] TS ad96c66d357843d0b6e9618a6f03eb36 (127.12.12.3:39473): tablet 10e7fc01cda74606a55c2da9a3c9a345 (table test-table [id=dbc7dc57331144dcaf49757f7e3ac153]) successfully deleted
I20260621 07:51:49.227169 14612 raft_consensus.cc:2258] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:49.227743 14568 tablet_bootstrap.cc:492] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9: No bootstrap required, opened a new log
I20260621 07:51:49.227841 14568 ts_tablet_manager.cc:1397] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260621 07:51:49.228278 14568 raft_consensus.cc:348] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:49.228387 14568 raft_consensus.cc:374] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:49.228415 14568 raft_consensus.cc:729] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8807cf54e8534190bd7936951af169e9, State: Initialized, Role: FOLLOWER
I20260621 07:51:49.228525 14568 consensus_queue.cc:260] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:49.228981 14568 ts_tablet_manager.cc:1428] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:51:49.229321 14612 ts_tablet_manager.cc:1905] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:49.232478 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:49.225657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:38061 (local address 127.12.12.3:42573)
0621 07:51:49.226895 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:49.226911 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:49.227009 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:49.227565 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:49.227568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:49.227586 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:49.227712 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:49.227718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.229605 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.229609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:49.230915 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:49.230928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.231987 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.231992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:49.232110 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:49.232139 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 07:51:49.232216 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 07:51:49.232301 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"thread_start_us":115,"threads_started":1}
I20260621 07:51:49.233280 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:49.226378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:42573 (local address 127.12.12.1:38061)
0621 07:51:49.227218 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:49.227225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:49.227242 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:51:49.227299 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:49.227302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:49.227353 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:49.227435 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:49.227847 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.228609 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.231190 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.231781 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.232011 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:49.232117 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:49.233042 (+ 925us) server_negotiation.cc:299] Negotiation successful
0621 07:51:49.233134 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":111,"threads_started":1}
I20260621 07:51:49.233893 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa198344cc04e6a9d8d7301e1932573" candidate_uuid: "ad96c66d357843d0b6e9618a6f03eb36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" is_pre_election: true
I20260621 07:51:49.234300 14612 ts_tablet_manager.cc:1918] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:51:49.234436 14612 log.cc:1196] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/10e7fc01cda74606a55c2da9a3c9a345
W20260621 07:51:49.234706 14429 leader_election.cc:341] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 57ae6638eeaf4b3b974c4f8a5c4b15ae (127.12.12.1:38061): Illegal state: must be running to vote when last-logged opid is not known
I20260621 07:51:49.235374 14612 ts_tablet_manager.cc:1939] T 10e7fc01cda74606a55c2da9a3c9a345 P 8807cf54e8534190bd7936951af169e9: Deleting consensus metadata
I20260621 07:51:49.236366 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:49.226611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:48285 (local address 127.12.12.2:44483)
0621 07:51:49.226742 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:49.226747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:49.229460 (+ 2713us) server_negotiation.cc:407] Connection header received
0621 07:51:49.232233 (+ 2773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:49.232237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:49.232285 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:49.232333 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:49.232795 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.233522 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.235095 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.235714 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.235844 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:49.235915 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:49.236170 (+ 255us) server_negotiation.cc:299] Negotiation successful
0621 07:51:49.236248 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:51:49.236383 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:49.226018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:44483 (local address 127.12.12.3:48285)
0621 07:51:49.229359 (+ 3341us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:49.229377 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:49.231070 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:49.232464 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:49.232468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:49.232493 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:49.232652 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:49.232661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.233681 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.233684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:49.234929 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:49.234939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.235891 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.235896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:49.236008 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:49.236042 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 07:51:49.236152 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 07:51:49.236233 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3265,"thread_start_us":81,"threads_started":1}
I20260621 07:51:49.236580 14045 catalog_manager.cc:4841] TS 8807cf54e8534190bd7936951af169e9 (127.12.12.2:44483): tablet 10e7fc01cda74606a55c2da9a3c9a345 (table test-table [id=dbc7dc57331144dcaf49757f7e3ac153]) successfully deleted
I20260621 07:51:49.236884 14358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa198344cc04e6a9d8d7301e1932573" candidate_uuid: "ad96c66d357843d0b6e9618a6f03eb36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8807cf54e8534190bd7936951af169e9" is_pre_election: true
I20260621 07:51:49.237112 14358 raft_consensus.cc:2454] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad96c66d357843d0b6e9618a6f03eb36 in term 0.
I20260621 07:51:49.237464 14429 leader_election.cc:302] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8807cf54e8534190bd7936951af169e9, ad96c66d357843d0b6e9618a6f03eb36; no voters: 57ae6638eeaf4b3b974c4f8a5c4b15ae
I20260621 07:51:49.237775 14617 raft_consensus.cc:2781] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:51:49.237905 14617 raft_consensus.cc:482] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:51:49.237974 14617 raft_consensus.cc:3037] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:49.238057 14567 tablet_bootstrap.cc:492] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: No bootstrap required, opened a new log
I20260621 07:51:49.238157 14567 ts_tablet_manager.cc:1397] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260621 07:51:49.238659 14567 raft_consensus.cc:348] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:49.238791 14567 raft_consensus.cc:374] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:49.238826 14567 raft_consensus.cc:729] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ae6638eeaf4b3b974c4f8a5c4b15ae, State: Initialized, Role: FOLLOWER
I20260621 07:51:49.238950 14567 consensus_queue.cc:260] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:49.239274 14567 ts_tablet_manager.cc:1428] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:51:49.239353 14617 raft_consensus.cc:504] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:49.240082 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa198344cc04e6a9d8d7301e1932573" candidate_uuid: "ad96c66d357843d0b6e9618a6f03eb36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae"
I20260621 07:51:49.240193 14218 raft_consensus.cc:3037] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:49.241389 14218 raft_consensus.cc:2454] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad96c66d357843d0b6e9618a6f03eb36 in term 1.
I20260621 07:51:49.241829 14617 leader_election.cc:288] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [CANDIDATE]: Term 1 election: Requested vote from peers 57ae6638eeaf4b3b974c4f8a5c4b15ae (127.12.12.1:38061), 8807cf54e8534190bd7936951af169e9 (127.12.12.2:44483)
I20260621 07:51:49.241848 14429 leader_election.cc:302] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ae6638eeaf4b3b974c4f8a5c4b15ae, ad96c66d357843d0b6e9618a6f03eb36; no voters:
I20260621 07:51:49.242304 14617 raft_consensus.cc:2781] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:49.242508 14617 raft_consensus.cc:686] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [term 1 LEADER]: Becoming Leader. State: Replica: ad96c66d357843d0b6e9618a6f03eb36, State: Running, Role: LEADER
I20260621 07:51:49.242535 14358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa198344cc04e6a9d8d7301e1932573" candidate_uuid: "ad96c66d357843d0b6e9618a6f03eb36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8807cf54e8534190bd7936951af169e9"
I20260621 07:51:49.242669 14358 raft_consensus.cc:3037] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:49.242774 14617 consensus_queue.cc:237] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:49.243707 14358 raft_consensus.cc:2454] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad96c66d357843d0b6e9618a6f03eb36 in term 1.
I20260621 07:51:49.245915 14059 catalog_manager.cc:5495] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad96c66d357843d0b6e9618a6f03eb36 (127.12.12.3). New cstate: current_term: 1 leader_uuid: "ad96c66d357843d0b6e9618a6f03eb36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } health_report { overall_health: HEALTHY } } }
I20260621 07:51:49.267606 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:49.263507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.12.12.3:39473)
0621 07:51:49.263838 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:49.263848 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:49.263860 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:51:49.263937 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:49.263939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:49.263989 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:49.264116 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:49.264368 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.264983 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.265813 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.266058 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.266327 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:49.266675 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:49.267139 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:49.267266 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:49.267343 (+ 77us) server_negotiation.cc:299] Negotiation successful
0621 07:51:49.267398 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":138,"threads_started":1}
I20260621 07:51:49.279261 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:49.275518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:59315 (local address 127.12.12.2:44483)
0621 07:51:49.275697 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:49.275701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:49.275805 (+ 104us) server_negotiation.cc:407] Connection header received
0621 07:51:49.275948 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:49.275950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:49.275997 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:49.276084 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:49.276379 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.276910 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.278177 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.278391 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.278529 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:49.278860 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:49.278931 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:49.278984 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:49.279098 (+ 114us) server_negotiation.cc:299] Negotiation successful
0621 07:51:49.279148 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 07:51:49.279651 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:49.275357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:44483 (local address 127.12.12.3:59315)
0621 07:51:49.275712 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:49.275728 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:49.275844 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:49.276103 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:49.276106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:49.276114 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:51:49.276245 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:49.276251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.277021 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.277025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:49.278050 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:49.278061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.278559 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.278563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:49.278672 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:49.278727 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:51:49.279009 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:51:49.279015 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:51:49.279080 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 07:51:49.279161 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":131,"threads_started":1}
I20260621 07:51:49.286458 14218 raft_consensus.cc:1261] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 FOLLOWER]: Refusing update from remote peer ad96c66d357843d0b6e9618a6f03eb36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:49.286602 14358 raft_consensus.cc:1261] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9 [term 1 FOLLOWER]: Refusing update from remote peer ad96c66d357843d0b6e9618a6f03eb36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:49.286952 14575 consensus_queue.cc:1035] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:49.287549 14575 consensus_queue.cc:1035] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:49.306700 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:49.302832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:39473 (local address 127.12.12.2:53309)
0621 07:51:49.303012 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:49.303028 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:49.303101 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:49.303360 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:49.303363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:49.303371 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:51:49.303481 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:49.303491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.304255 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.304259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:49.305217 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:49.305227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.305792 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.305796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:49.305886 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:49.305919 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:51:49.306337 (+ 418us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:51:49.306343 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:51:49.306432 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 07:51:49.306531 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260621 07:51:49.307083 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:49.303019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:53309 (local address 127.12.12.3:39473)
0621 07:51:49.303145 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:49.303149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:49.303162 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:51:49.303207 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:49.303209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:49.303265 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:49.303348 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:49.303610 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.304151 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.305327 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:49.305655 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:49.305795 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:49.306057 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:49.306144 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:49.306212 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:49.306883 (+ 671us) server_negotiation.cc:299] Negotiation successful
0621 07:51:49.306941 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260621 07:51:49.307737 14596 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 07:51:50.879520 14061 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:50.879755 14061 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:50.882381 14061 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb: Sending DeleteTablet for 3 replicas of tablet 0aa198344cc04e6a9d8d7301e1932573
I20260621 07:51:50.883037 14198 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa198344cc04e6a9d8d7301e1932573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:50 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:44328
I20260621 07:51:50.883047 14478 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa198344cc04e6a9d8d7301e1932573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:50 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:41748
I20260621 07:51:50.883059 14338 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa198344cc04e6a9d8d7301e1932573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:50 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:38968
I20260621 07:51:50.883437 14656 tablet_replica.cc:330] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9: stopping tablet replica
I20260621 07:51:50.883643 14656 raft_consensus.cc:2229] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:50.883783 14656 raft_consensus.cc:2258] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:50.884023 14655 tablet_replica.cc:330] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36: stopping tablet replica
I20260621 07:51:50.884172 14655 raft_consensus.cc:2229] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:51:50.884475 14655 raft_consensus.cc:2258] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:50.885283 14656 ts_tablet_manager.cc:1905] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:50.885675 14654 tablet_replica.cc:330] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: stopping tablet replica
I20260621 07:51:50.885834 14654 raft_consensus.cc:2229] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:50.885980 14654 raft_consensus.cc:2258] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:50.886040 14655 ts_tablet_manager.cc:1905] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:50.886809 14654 ts_tablet_manager.cc:1905] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:50.889364 14656 ts_tablet_manager.cc:1918] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 07:51:50.889453 14656 log.cc:1196] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/0aa198344cc04e6a9d8d7301e1932573
I20260621 07:51:50.889565 14654 ts_tablet_manager.cc:1918] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 07:51:50.889632 14654 log.cc:1196] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/wal/wals/0aa198344cc04e6a9d8d7301e1932573
I20260621 07:51:50.889740 14656 ts_tablet_manager.cc:1939] T 0aa198344cc04e6a9d8d7301e1932573 P 8807cf54e8534190bd7936951af169e9: Deleting consensus metadata
I20260621 07:51:50.889894 14654 ts_tablet_manager.cc:1939] T 0aa198344cc04e6a9d8d7301e1932573 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Deleting consensus metadata
I20260621 07:51:50.890591 14045 catalog_manager.cc:4841] TS 8807cf54e8534190bd7936951af169e9 (127.12.12.2:44483): tablet 0aa198344cc04e6a9d8d7301e1932573 (table test-table [id=9ab225923cf94938bb90a3113a0d8ec2]) successfully deleted
I20260621 07:51:50.890643 14655 ts_tablet_manager.cc:1918] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 07:51:50.890811 14655 log.cc:1196] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/0aa198344cc04e6a9d8d7301e1932573
I20260621 07:51:50.890810 14045 catalog_manager.cc:4841] TS 57ae6638eeaf4b3b974c4f8a5c4b15ae (127.12.12.1:38061): tablet 0aa198344cc04e6a9d8d7301e1932573 (table test-table [id=9ab225923cf94938bb90a3113a0d8ec2]) successfully deleted
I20260621 07:51:50.891098 14655 ts_tablet_manager.cc:1939] T 0aa198344cc04e6a9d8d7301e1932573 P ad96c66d357843d0b6e9618a6f03eb36: Deleting consensus metadata
I20260621 07:51:50.891782 14047 catalog_manager.cc:4841] TS ad96c66d357843d0b6e9618a6f03eb36 (127.12.12.3:39473): tablet 0aa198344cc04e6a9d8d7301e1932573 (table test-table [id=9ab225923cf94938bb90a3113a0d8ec2]) successfully deleted
Jun 21 07:51:50 dist-test-slave-lbpt krb5kdc[14016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 07:51:50 dist-test-slave-lbpt krb5kdc[14016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:50.918931 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:50.904096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.12.12.62:42395)
0621 07:51:50.904430 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:50.904437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:50.904452 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:51:50.904578 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:50.904580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:50.904624 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:50.904715 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:50.905453 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:50.906042 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:50.906964 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:50.907207 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:50.907303 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:50.915145 (+ 7842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:50.915149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:50.915151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:50.915185 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:50.916695 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:50.917164 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:50.917167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:50.917169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:50.917265 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:50.917703 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:50.917710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:50.917713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:50.918035 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:50.918175 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:50.918604 (+ 429us) server_negotiation.cc:299] Negotiation successful
0621 07:51:50.918776 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":153,"threads_started":1}
I20260621 07:51:50.921921 14061 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:51:50.922448 14061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:51:50.927464 14198 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc4ff0976ad4e588ecd469433f59145 (DEFAULT_TABLE table=test-table [id=72cbb11ad07a4a949d5a85e03dd5eb52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:50.927520 14338 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc4ff0976ad4e588ecd469433f59145 (DEFAULT_TABLE table=test-table [id=72cbb11ad07a4a949d5a85e03dd5eb52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:50.927780 14198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc4ff0976ad4e588ecd469433f59145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:50.927721 14478 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc4ff0976ad4e588ecd469433f59145 (DEFAULT_TABLE table=test-table [id=72cbb11ad07a4a949d5a85e03dd5eb52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:50.927812 14338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc4ff0976ad4e588ecd469433f59145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:50.927968 14478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc4ff0976ad4e588ecd469433f59145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:50.929870 14665 tablet_bootstrap.cc:492] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9: Bootstrap starting.
I20260621 07:51:50.930361 14666 tablet_bootstrap.cc:492] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36: Bootstrap starting.
I20260621 07:51:50.930933 14665 tablet_bootstrap.cc:654] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:50.931263 14666 tablet_bootstrap.cc:654] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:50.931931 14667 tablet_bootstrap.cc:492] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Bootstrap starting.
I20260621 07:51:50.932281 14665 tablet_bootstrap.cc:492] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9: No bootstrap required, opened a new log
I20260621 07:51:50.932396 14665 ts_tablet_manager.cc:1397] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 07:51:50.932476 14666 tablet_bootstrap.cc:492] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36: No bootstrap required, opened a new log
I20260621 07:51:50.932569 14666 ts_tablet_manager.cc:1397] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:51:50.932811 14667 tablet_bootstrap.cc:654] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:50.932869 14665 raft_consensus.cc:348] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:50.932992 14665 raft_consensus.cc:374] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:50.933027 14665 raft_consensus.cc:729] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8807cf54e8534190bd7936951af169e9, State: Initialized, Role: FOLLOWER
I20260621 07:51:50.933073 14666 raft_consensus.cc:348] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:50.933187 14666 raft_consensus.cc:374] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:50.933228 14666 raft_consensus.cc:729] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad96c66d357843d0b6e9618a6f03eb36, State: Initialized, Role: FOLLOWER
I20260621 07:51:50.933405 14666 consensus_queue.cc:260] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:50.933511 14665 consensus_queue.cc:260] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:50.933959 14667 tablet_bootstrap.cc:492] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: No bootstrap required, opened a new log
I20260621 07:51:50.934046 14667 ts_tablet_manager.cc:1397] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:51:50.934172 14665 ts_tablet_manager.cc:1428] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:50.934438 14666 ts_tablet_manager.cc:1428] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:50.934514 14667 raft_consensus.cc:348] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:50.934618 14667 raft_consensus.cc:374] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:50.934649 14667 raft_consensus.cc:729] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ae6638eeaf4b3b974c4f8a5c4b15ae, State: Initialized, Role: FOLLOWER
I20260621 07:51:50.934777 14667 consensus_queue.cc:260] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:50.935140 14667 ts_tablet_manager.cc:1428] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:51:50.996248 14671 raft_consensus.cc:482] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:51:50.996384 14671 raft_consensus.cc:504] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:50.997428 14671 leader_election.cc:288] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8807cf54e8534190bd7936951af169e9 (127.12.12.2:44483), ad96c66d357843d0b6e9618a6f03eb36 (127.12.12.3:39473)
I20260621 07:51:51.002712 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:50.997968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:54005 (local address 127.12.12.2:44483)
0621 07:51:50.998239 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:50.998252 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:50.998272 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:51:50.998331 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:50.998334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:50.998387 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:50.998499 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:50.998780 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:50.999610 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.001264 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:51.001898 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.002024 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:51.002101 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:51.002464 (+ 363us) server_negotiation.cc:299] Negotiation successful
0621 07:51:51.002585 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":89,"threads_started":1}
I20260621 07:51:51.002727 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:50.997888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:35931 (local address 127.12.12.3:39473)
0621 07:51:50.998544 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:50.998553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:50.998567 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:51:50.998626 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:50.998628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:50.998672 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:50.998772 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:50.999014 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:50.999801 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.001264 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:51.001922 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.002075 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:51.002136 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:51.002475 (+ 339us) server_negotiation.cc:299] Negotiation successful
0621 07:51:51.002552 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":118,"threads_started":1}
I20260621 07:51:51.002728 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:50.997683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:39473 (local address 127.12.12.1:35931)
0621 07:51:50.998211 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:50.998226 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:50.998330 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:50.998807 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:50.998809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:50.998823 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:50.998908 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:50.998912 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:50.999903 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:50.999905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:51.001128 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:51.001141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:51.002090 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.002092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:51.002261 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:51.002325 (+ 64us) client_negotiation.cc:769] Sending connection context
0621 07:51:51.002399 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 07:51:51.002518 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":125,"thread_start_us":126,"threads_started":1}
I20260621 07:51:51.002732 14672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:50.997683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:44483 (local address 127.12.12.1:54005)
0621 07:51:50.998101 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:50.998119 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:50.998233 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:50.998526 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:50.998528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:50.998543 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:50.998666 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:50.998671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:50.999717 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:50.999720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:51.001143 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:51.001150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:51.002087 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.002091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:51.002263 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:51.002319 (+ 56us) client_negotiation.cc:769] Sending connection context
0621 07:51:51.002395 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 07:51:51.002518 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":134,"thread_start_us":131,"threads_started":1}
I20260621 07:51:51.003283 14498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc4ff0976ad4e588ecd469433f59145" candidate_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad96c66d357843d0b6e9618a6f03eb36" is_pre_election: true
I20260621 07:51:51.003515 14498 raft_consensus.cc:2454] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57ae6638eeaf4b3b974c4f8a5c4b15ae in term 0.
I20260621 07:51:51.003806 14358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc4ff0976ad4e588ecd469433f59145" candidate_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8807cf54e8534190bd7936951af169e9" is_pre_election: true
I20260621 07:51:51.003945 14358 raft_consensus.cc:2454] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57ae6638eeaf4b3b974c4f8a5c4b15ae in term 0.
I20260621 07:51:51.003976 14151 leader_election.cc:302] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ae6638eeaf4b3b974c4f8a5c4b15ae, ad96c66d357843d0b6e9618a6f03eb36; no voters:
I20260621 07:51:51.004287 14671 raft_consensus.cc:2781] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:51:51.004343 14671 raft_consensus.cc:482] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:51:51.004370 14671 raft_consensus.cc:3037] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:51.005368 14671 raft_consensus.cc:504] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:51.005729 14671 leader_election.cc:288] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [CANDIDATE]: Term 1 election: Requested vote from peers 8807cf54e8534190bd7936951af169e9 (127.12.12.2:44483), ad96c66d357843d0b6e9618a6f03eb36 (127.12.12.3:39473)
I20260621 07:51:51.006076 14498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc4ff0976ad4e588ecd469433f59145" candidate_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad96c66d357843d0b6e9618a6f03eb36"
I20260621 07:51:51.006209 14498 raft_consensus.cc:3037] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:51.006217 14358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc4ff0976ad4e588ecd469433f59145" candidate_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8807cf54e8534190bd7936951af169e9"
I20260621 07:51:51.006338 14358 raft_consensus.cc:3037] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:51.007232 14498 raft_consensus.cc:2454] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57ae6638eeaf4b3b974c4f8a5c4b15ae in term 1.
I20260621 07:51:51.007259 14358 raft_consensus.cc:2454] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57ae6638eeaf4b3b974c4f8a5c4b15ae in term 1.
I20260621 07:51:51.007511 14151 leader_election.cc:302] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ae6638eeaf4b3b974c4f8a5c4b15ae, ad96c66d357843d0b6e9618a6f03eb36; no voters:
I20260621 07:51:51.007692 14671 raft_consensus.cc:2781] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:51.007899 14671 raft_consensus.cc:686] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 LEADER]: Becoming Leader. State: Replica: 57ae6638eeaf4b3b974c4f8a5c4b15ae, State: Running, Role: LEADER
I20260621 07:51:51.008165 14671 consensus_queue.cc:237] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:51.010612 14061 catalog_manager.cc:5495] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 57ae6638eeaf4b3b974c4f8a5c4b15ae (127.12.12.1). New cstate: current_term: 1 leader_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } health_report { overall_health: UNKNOWN } } }
I20260621 07:51:51.043728 14679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:51.040085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.12.12.1:38061)
0621 07:51:51.040337 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:51.040343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:51.040355 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:51:51.040426 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:51.040428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:51.040521 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:51.040647 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:51.040857 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:51.041410 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.042253 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:51.042502 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.042581 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:51.042852 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:51.043300 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:51.043384 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:51.043443 (+ 59us) server_negotiation.cc:299] Negotiation successful
0621 07:51:51.043488 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":113,"threads_started":1}
I20260621 07:51:51.053833 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:51.050840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:33235 (local address 127.12.12.2:44483)
0621 07:51:51.050990 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:51.050995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:51.051010 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:51:51.051059 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:51.051061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:51.051105 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:51.051176 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:51.051398 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:51.051886 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.052881 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:51.053096 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.053191 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:51.053449 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:51.053520 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:51.053576 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:51.053670 (+ 94us) server_negotiation.cc:299] Negotiation successful
0621 07:51:51.053720 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:51:51.053851 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:51.050557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:44483 (local address 127.12.12.1:33235)
0621 07:51:51.050871 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:51.050884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:51.050993 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:51.051194 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:51.051196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:51.051202 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:51:51.051309 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:51.051313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:51.052000 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.052003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:51.052767 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:51.052777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:51.053208 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.053211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:51.053283 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:51.053326 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:51:51.053598 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:51:51.053604 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:51:51.053660 (+ 56us) client_negotiation.cc:240] Negotiation successful
0621 07:51:51.053729 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
I20260621 07:51:51.060176 14498 raft_consensus.cc:1261] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Refusing update from remote peer 57ae6638eeaf4b3b974c4f8a5c4b15ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:51.060177 14358 raft_consensus.cc:1261] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9 [term 1 FOLLOWER]: Refusing update from remote peer 57ae6638eeaf4b3b974c4f8a5c4b15ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:51.060652 14676 consensus_queue.cc:1035] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:51.060845 14671 consensus_queue.cc:1035] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:51.079038 14679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:51.075730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:54467 (local address 127.12.12.1:38061)
0621 07:51:51.075945 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:51.075952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:51.075998 (+ 46us) server_negotiation.cc:407] Connection header received
0621 07:51:51.076158 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:51.076162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:51.076208 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:51.076297 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:51.076536 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:51.077068 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.077931 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:51.078177 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.078288 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:51.078594 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:51.078678 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:51.078759 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:51.078851 (+ 92us) server_negotiation.cc:299] Negotiation successful
0621 07:51:51.078901 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260621 07:51:51.079046 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:51.075668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:38061 (local address 127.12.12.2:54467)
0621 07:51:51.075922 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:51.075938 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:51.076044 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:51.076308 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:51.076311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:51.076319 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:51:51.076427 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:51.076432 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:51.077183 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.077187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:51.077808 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:51.077816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:51.078289 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.078293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:51.078381 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:51.078464 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:51:51.078763 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:51:51.078771 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 07:51:51.078832 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 07:51:51.078918 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
I20260621 07:51:51.101631 14059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:51.101841 14059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:51.104249 14059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb: Sending DeleteTablet for 3 replicas of tablet 4dc4ff0976ad4e588ecd469433f59145
I20260621 07:51:51.104863 14198 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc4ff0976ad4e588ecd469433f59145 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:51 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:44328
I20260621 07:51:51.104912 14338 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc4ff0976ad4e588ecd469433f59145 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:51 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:38968
I20260621 07:51:51.104964 14478 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc4ff0976ad4e588ecd469433f59145 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:51 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:41748
I20260621 07:51:51.105059 14654 tablet_replica.cc:330] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: stopping tablet replica
I20260621 07:51:51.105154 14655 tablet_replica.cc:330] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36: stopping tablet replica
I20260621 07:51:51.105195 14654 raft_consensus.cc:2229] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:51:51.105275 14655 raft_consensus.cc:2229] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:51.105404 14655 raft_consensus.cc:2258] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:51.105531 14654 raft_consensus.cc:2258] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:51.105752 14656 tablet_replica.cc:330] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9: stopping tablet replica
I20260621 07:51:51.105885 14656 raft_consensus.cc:2229] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:51.106016 14656 raft_consensus.cc:2258] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:51.106784 14655 ts_tablet_manager.cc:1905] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:51.106931 14654 ts_tablet_manager.cc:1905] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:51.109014 14654 ts_tablet_manager.cc:1918] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:51:51.109117 14654 log.cc:1196] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-0/wal/wals/4dc4ff0976ad4e588ecd469433f59145
I20260621 07:51:51.109356 14654 ts_tablet_manager.cc:1939] T 4dc4ff0976ad4e588ecd469433f59145 P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Deleting consensus metadata
I20260621 07:51:51.109592 14655 ts_tablet_manager.cc:1918] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:51:51.109674 14655 log.cc:1196] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/4dc4ff0976ad4e588ecd469433f59145
I20260621 07:51:51.109934 14655 ts_tablet_manager.cc:1939] T 4dc4ff0976ad4e588ecd469433f59145 P ad96c66d357843d0b6e9618a6f03eb36: Deleting consensus metadata
I20260621 07:51:51.110589 14047 catalog_manager.cc:4841] TS ad96c66d357843d0b6e9618a6f03eb36 (127.12.12.3:39473): tablet 4dc4ff0976ad4e588ecd469433f59145 (table test-table [id=72cbb11ad07a4a949d5a85e03dd5eb52]) successfully deleted
I20260621 07:51:51.110736 14045 catalog_manager.cc:4841] TS 57ae6638eeaf4b3b974c4f8a5c4b15ae (127.12.12.1:38061): tablet 4dc4ff0976ad4e588ecd469433f59145 (table test-table [id=72cbb11ad07a4a949d5a85e03dd5eb52]) successfully deleted
I20260621 07:51:51.110901 14059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:51:51.111342 14059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:51:51.112433 14656 ts_tablet_manager.cc:1905] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:51.115062 14656 ts_tablet_manager.cc:1918] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:51:51.115156 14656 log.cc:1196] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/4dc4ff0976ad4e588ecd469433f59145
I20260621 07:51:51.115384 14656 ts_tablet_manager.cc:1939] T 4dc4ff0976ad4e588ecd469433f59145 P 8807cf54e8534190bd7936951af169e9: Deleting consensus metadata
I20260621 07:51:51.116078 14198 tablet_service.cc:1467] Processing CreateTablet for tablet 102105dc33c8468fa523def4699da3ad (DEFAULT_TABLE table=test-table [id=b51dccf7151a42b1906fd31423c9c509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:51.116179 14478 tablet_service.cc:1467] Processing CreateTablet for tablet 102105dc33c8468fa523def4699da3ad (DEFAULT_TABLE table=test-table [id=b51dccf7151a42b1906fd31423c9c509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:51.116381 14198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 102105dc33c8468fa523def4699da3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:51.116428 14478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 102105dc33c8468fa523def4699da3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:51.118168 14667 tablet_bootstrap.cc:492] T 102105dc33c8468fa523def4699da3ad P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Bootstrap starting.
I20260621 07:51:51.118885 14666 tablet_bootstrap.cc:492] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36: Bootstrap starting.
I20260621 07:51:51.119055 14045 catalog_manager.cc:4841] TS 8807cf54e8534190bd7936951af169e9 (127.12.12.2:44483): tablet 4dc4ff0976ad4e588ecd469433f59145 (table test-table [id=72cbb11ad07a4a949d5a85e03dd5eb52]) successfully deleted
I20260621 07:51:51.119210 14667 tablet_bootstrap.cc:654] T 102105dc33c8468fa523def4699da3ad P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:51.119653 14666 tablet_bootstrap.cc:654] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:51.119987 14338 tablet_service.cc:1467] Processing CreateTablet for tablet 102105dc33c8468fa523def4699da3ad (DEFAULT_TABLE table=test-table [id=b51dccf7151a42b1906fd31423c9c509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:51.120232 14338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 102105dc33c8468fa523def4699da3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:51.122078 14665 tablet_bootstrap.cc:492] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9: Bootstrap starting.
I20260621 07:51:51.122658 14667 tablet_bootstrap.cc:492] T 102105dc33c8468fa523def4699da3ad P 57ae6638eeaf4b3b974c4f8a5c4b15ae: No bootstrap required, opened a new log
I20260621 07:51:51.122746 14667 ts_tablet_manager.cc:1397] T 102105dc33c8468fa523def4699da3ad P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260621 07:51:51.122918 14665 tablet_bootstrap.cc:654] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:51.123090 14666 tablet_bootstrap.cc:492] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36: No bootstrap required, opened a new log
I20260621 07:51:51.123162 14666 ts_tablet_manager.cc:1397] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260621 07:51:51.123227 14667 raft_consensus.cc:348] T 102105dc33c8468fa523def4699da3ad P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:51.123358 14667 raft_consensus.cc:374] T 102105dc33c8468fa523def4699da3ad P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:51.123394 14667 raft_consensus.cc:729] T 102105dc33c8468fa523def4699da3ad P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ae6638eeaf4b3b974c4f8a5c4b15ae, State: Initialized, Role: FOLLOWER
I20260621 07:51:51.123512 14666 raft_consensus.cc:348] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:51.123523 14667 consensus_queue.cc:260] T 102105dc33c8468fa523def4699da3ad P 57ae6638eeaf4b3b974c4f8a5c4b15ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:51.123627 14666 raft_consensus.cc:374] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:51.123658 14666 raft_consensus.cc:729] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad96c66d357843d0b6e9618a6f03eb36, State: Initialized, Role: FOLLOWER
I20260621 07:51:51.123767 14666 consensus_queue.cc:260] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:51.124001 14665 tablet_bootstrap.cc:492] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9: No bootstrap required, opened a new log
I20260621 07:51:51.124019 14666 ts_tablet_manager.cc:1428] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:51:51.124074 14665 ts_tablet_manager.cc:1397] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:51:51.124491 14665 raft_consensus.cc:348] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:51.124594 14665 raft_consensus.cc:374] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:51.124622 14665 raft_consensus.cc:729] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8807cf54e8534190bd7936951af169e9, State: Initialized, Role: FOLLOWER
I20260621 07:51:51.123842 14667 ts_tablet_manager.cc:1428] T 102105dc33c8468fa523def4699da3ad P 57ae6638eeaf4b3b974c4f8a5c4b15ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:51:51.124738 14665 consensus_queue.cc:260] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:51.125020 14665 ts_tablet_manager.cc:1428] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:51:51.126358 14635 raft_consensus.cc:482] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:51:51.126493 14635 raft_consensus.cc:504] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:51.126852 14635 leader_election.cc:288] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57ae6638eeaf4b3b974c4f8a5c4b15ae (127.12.12.1:38061), ad96c66d357843d0b6e9618a6f03eb36 (127.12.12.3:39473)
I20260621 07:51:51.127167 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102105dc33c8468fa523def4699da3ad" candidate_uuid: "8807cf54e8534190bd7936951af169e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" is_pre_election: true
I20260621 07:51:51.127282 14218 raft_consensus.cc:2454] T 102105dc33c8468fa523def4699da3ad P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8807cf54e8534190bd7936951af169e9 in term 0.
I20260621 07:51:51.127563 14289 leader_election.cc:302] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ae6638eeaf4b3b974c4f8a5c4b15ae, 8807cf54e8534190bd7936951af169e9; no voters:
I20260621 07:51:51.127777 14635 raft_consensus.cc:2781] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:51:51.127832 14635 raft_consensus.cc:482] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:51:51.127868 14635 raft_consensus.cc:3037] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:51.128934 14635 raft_consensus.cc:504] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:51.129267 14635 leader_election.cc:288] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [CANDIDATE]: Term 1 election: Requested vote from peers 57ae6638eeaf4b3b974c4f8a5c4b15ae (127.12.12.1:38061), ad96c66d357843d0b6e9618a6f03eb36 (127.12.12.3:39473)
I20260621 07:51:51.130157 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102105dc33c8468fa523def4699da3ad" candidate_uuid: "8807cf54e8534190bd7936951af169e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae"
I20260621 07:51:51.130267 14218 raft_consensus.cc:3037] T 102105dc33c8468fa523def4699da3ad P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:51.130360 14498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102105dc33c8468fa523def4699da3ad" candidate_uuid: "8807cf54e8534190bd7936951af169e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad96c66d357843d0b6e9618a6f03eb36" is_pre_election: true
I20260621 07:51:51.130532 14498 raft_consensus.cc:2454] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8807cf54e8534190bd7936951af169e9 in term 0.
I20260621 07:51:51.130360 14497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102105dc33c8468fa523def4699da3ad" candidate_uuid: "8807cf54e8534190bd7936951af169e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad96c66d357843d0b6e9618a6f03eb36"
I20260621 07:51:51.130738 14497 raft_consensus.cc:3037] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:51.131371 14218 raft_consensus.cc:2454] T 102105dc33c8468fa523def4699da3ad P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8807cf54e8534190bd7936951af169e9 in term 1.
I20260621 07:51:51.131680 14289 leader_election.cc:302] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ae6638eeaf4b3b974c4f8a5c4b15ae, 8807cf54e8534190bd7936951af169e9; no voters:
I20260621 07:51:51.131803 14497 raft_consensus.cc:2454] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8807cf54e8534190bd7936951af169e9 in term 1.
I20260621 07:51:51.131976 14635 raft_consensus.cc:2781] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:51.132064 14635 raft_consensus.cc:686] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [term 1 LEADER]: Becoming Leader. State: Replica: 8807cf54e8534190bd7936951af169e9, State: Running, Role: LEADER
I20260621 07:51:51.132198 14635 consensus_queue.cc:237] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } }
I20260621 07:51:51.133719 14059 catalog_manager.cc:5495] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8807cf54e8534190bd7936951af169e9 (127.12.12.2). New cstate: current_term: 1 leader_uuid: "8807cf54e8534190bd7936951af169e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8807cf54e8534190bd7936951af169e9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 } health_report { overall_health: UNKNOWN } } }
I20260621 07:51:51.150352 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:51.147049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.12.12.2:44483)
0621 07:51:51.147158 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:51.147165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:51.147182 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:51:51.147353 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:51.147356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:51.147407 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:51.147492 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:51.147749 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:51.148270 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.149255 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:51.149482 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:51.149591 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:51.149848 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:51.149981 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:51.150059 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:51.150159 (+ 100us) server_negotiation.cc:299] Negotiation successful
0621 07:51:51.150208 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 07:51:51.157023 14497 raft_consensus.cc:1261] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Refusing update from remote peer 8807cf54e8534190bd7936951af169e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:51.157413 14218 raft_consensus.cc:1261] T 102105dc33c8468fa523def4699da3ad P 57ae6638eeaf4b3b974c4f8a5c4b15ae [term 1 FOLLOWER]: Refusing update from remote peer 8807cf54e8534190bd7936951af169e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:51.157455 14640 consensus_queue.cc:1035] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad96c66d357843d0b6e9618a6f03eb36" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:51.157903 14640 consensus_queue.cc:1035] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57ae6638eeaf4b3b974c4f8a5c4b15ae" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 07:51:51.171921 14633 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 07:51:52.722813 14060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:52.722988 14060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:52.725538 14060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8899dd7bde54d85888f5bbb9b467fbb: Sending DeleteTablet for 3 replicas of tablet 102105dc33c8468fa523def4699da3ad
I20260621 07:51:52.726200 14198 tablet_service.cc:1514] Processing DeleteTablet for tablet 102105dc33c8468fa523def4699da3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:52 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:44328
I20260621 07:51:52.726203 14337 tablet_service.cc:1514] Processing DeleteTablet for tablet 102105dc33c8468fa523def4699da3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:52 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:38968
I20260621 07:51:52.726601 14715 tablet_replica.cc:330] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9: stopping tablet replica
I20260621 07:51:52.726614 14478 tablet_service.cc:1514] Processing DeleteTablet for tablet 102105dc33c8468fa523def4699da3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:52 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:41748
I20260621 07:51:52.726682 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 14133
I20260621 07:51:52.726743 14715 raft_consensus.cc:2229] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:51:52.727596 14715 raft_consensus.cc:2258] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:52.727943 14717 tablet_replica.cc:330] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36: stopping tablet replica
I20260621 07:51:52.728104 14717 raft_consensus.cc:2229] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:52.728228 14717 raft_consensus.cc:2258] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:52.728334 14715 ts_tablet_manager.cc:1905] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:52.729111 14717 ts_tablet_manager.cc:1905] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:52.731796 14717 ts_tablet_manager.cc:1918] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 07:51:52.731899 14717 log.cc:1196] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/102105dc33c8468fa523def4699da3ad
I20260621 07:51:52.731902 14715 ts_tablet_manager.cc:1918] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 07:51:52.731977 14715 log.cc:1196] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/102105dc33c8468fa523def4699da3ad
I20260621 07:51:52.732205 14717 ts_tablet_manager.cc:1939] T 102105dc33c8468fa523def4699da3ad P ad96c66d357843d0b6e9618a6f03eb36: Deleting consensus metadata
I20260621 07:51:52.732290 14715 ts_tablet_manager.cc:1939] T 102105dc33c8468fa523def4699da3ad P 8807cf54e8534190bd7936951af169e9: Deleting consensus metadata
I20260621 07:51:52.732946 14047 catalog_manager.cc:4841] TS ad96c66d357843d0b6e9618a6f03eb36 (127.12.12.3:39473): tablet 102105dc33c8468fa523def4699da3ad (table test-table [id=b51dccf7151a42b1906fd31423c9c509]) successfully deleted
I20260621 07:51:52.733021 14045 catalog_manager.cc:4841] TS 8807cf54e8534190bd7936951af169e9 (127.12.12.2:44483): tablet 102105dc33c8468fa523def4699da3ad (table test-table [id=b51dccf7151a42b1906fd31423c9c509]) successfully deleted
I20260621 07:51:52.735428 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 14273
W20260621 07:51:52.735435 14045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.1:38061 (error 108)
I20260621 07:51:52.737083 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:52.736577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:38061 (local address 127.0.0.1:36702)
0621 07:51:52.736853 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:52.736947 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:38061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
W20260621 07:51:52.737244 14045 catalog_manager.cc:4584] TS 57ae6638eeaf4b3b974c4f8a5c4b15ae (127.12.12.1:38061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 102105dc33c8468fa523def4699da3ad: Network error: Client connection negotiation failed: client connection to 127.12.12.1:38061: connect: Connection refused (error 111)
I20260621 07:51:52.744362 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 14413
I20260621 07:51:52.751147 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 14029
2026-06-21T07:51:52Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6439 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:52 dist-test-slave-lbpt krb5kdc[14722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:52 dist-test-slave-lbpt krb5kdc[14722](info): set up 2 sockets
Jun 21 07:51:52 dist-test-slave-lbpt krb5kdc[14722](info): commencing operation
krb5kdc: starting...
W20260621 07:51:53.798308 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:51:53 dist-test-slave-lbpt krb5kdc[14722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:51:53.919406 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:36171
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:36171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:54.032485 14735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:54.032822 14735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:54.032896 14735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:54.037019 14735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:54.037137 14735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:54.037177 14735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:54.037215 14735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:54.037258 14735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:54.042847 14735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:36171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:36171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:54.044085 14735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:54.045190 14735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:54.054335 14741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:54.054330 14743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:54.054330 14740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:54.054401 14735 server_base.cc:1034] running on GCE node
I20260621 07:51:54.055231 14735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:54.056370 14735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:54.057550 14735 hybrid_clock.cc:648] HybridClock initialized: now 1782028314057537 us; error 27 us; skew 500 ppm
Jun 21 07:51:54 dist-test-slave-lbpt krb5kdc[14722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028314, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:54.060758 14735 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:51:54.061705 14735 webserver.cc:460] Webserver started at http://127.12.12.62:33119/ using document root <none> and password file <none>
I20260621 07:51:54.062331 14735 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:54.062397 14735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:54.062687 14735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:54.064800 14735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "6b7c0bd52e224279a80fe020bce3b23e"
format_stamp: "Formatted at 2026-06-21 07:51:54 on dist-test-slave-lbpt"
server_key: "bf0b1d59f393a3ed6688ddd180b62398"
server_key_iv: "dece63f4b5064aaef7b5df7af45b876d"
server_key_version: "encryptionkey@0"
I20260621 07:51:54.065430 14735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "6b7c0bd52e224279a80fe020bce3b23e"
format_stamp: "Formatted at 2026-06-21 07:51:54 on dist-test-slave-lbpt"
server_key: "bf0b1d59f393a3ed6688ddd180b62398"
server_key_iv: "dece63f4b5064aaef7b5df7af45b876d"
server_key_version: "encryptionkey@0"
I20260621 07:51:54.069275 14735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260621 07:51:54.071847 14750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:54.073120 14735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:51:54.073268 14735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "6b7c0bd52e224279a80fe020bce3b23e"
format_stamp: "Formatted at 2026-06-21 07:51:54 on dist-test-slave-lbpt"
server_key: "bf0b1d59f393a3ed6688ddd180b62398"
server_key_iv: "dece63f4b5064aaef7b5df7af45b876d"
server_key_version: "encryptionkey@0"
I20260621 07:51:54.073381 14735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:54.094920 14735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:54.099071 14735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:54.099324 14735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:54.107648 14735 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:36171
I20260621 07:51:54.107658 14802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:36171 every 8 connection(s)
I20260621 07:51:54.108771 14735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:54.111795 14803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:54.116030 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 14735
I20260621 07:51:54.116135 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:51:54.116413 12336 external_mini_cluster.cc:1428] Setting key 95213773d9b989c74ca2f7fbaa9c09b2
I20260621 07:51:54.118052 14803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e: Bootstrap starting.
I20260621 07:51:54.120797 14803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:54.121592 14803 log.cc:824] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:54.124017 14803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e: No bootstrap required, opened a new log
Jun 21 07:51:54 dist-test-slave-lbpt krb5kdc[14722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:54.127943 14803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c0bd52e224279a80fe020bce3b23e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36171 } }
I20260621 07:51:54.128216 14803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:54.128306 14803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7c0bd52e224279a80fe020bce3b23e, State: Initialized, Role: FOLLOWER
I20260621 07:51:54.128876 14803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c0bd52e224279a80fe020bce3b23e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36171 } }
I20260621 07:51:54.129069 14803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:54.129150 14803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:54.129264 14803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:54.130551 14803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c0bd52e224279a80fe020bce3b23e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36171 } }
I20260621 07:51:54.130954 14803 leader_election.cc:302] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b7c0bd52e224279a80fe020bce3b23e; no voters:
I20260621 07:51:54.131335 14803 leader_election.cc:288] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:54.131445 14808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:54.131744 14808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [term 1 LEADER]: Becoming Leader. State: Replica: 6b7c0bd52e224279a80fe020bce3b23e, State: Running, Role: LEADER
I20260621 07:51:54.132102 14808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c0bd52e224279a80fe020bce3b23e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36171 } }
I20260621 07:51:54.132781 14803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:54.132921 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:54.117843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.12.12.62:36171)
0621 07:51:54.118270 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:54.118286 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:54.118322 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:51:54.119036 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:54.119054 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:54.119394 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:54.119696 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:54.120603 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:54.121696 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:54.123130 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:54.123590 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:54.123717 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:54.126723 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:54.126727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:54.126742 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:54.126768 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:54.129446 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:54.129897 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:54.129904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:54.129910 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:54.130070 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:54.130355 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:54.130362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:54.130367 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:54.131005 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:54.131206 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:54.131566 (+ 360us) server_negotiation.cc:299] Negotiation successful
0621 07:51:54.131895 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":139,"threads_started":1}
I20260621 07:51:54.133213 14808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b7c0bd52e224279a80fe020bce3b23e. Latest consensus state: current_term: 1 leader_uuid: "6b7c0bd52e224279a80fe020bce3b23e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c0bd52e224279a80fe020bce3b23e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36171 } } }
I20260621 07:51:54.133333 14808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:54.133867 14810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b7c0bd52e224279a80fe020bce3b23e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c0bd52e224279a80fe020bce3b23e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36171 } } }
I20260621 07:51:54.134016 14810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:54.134327 14812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:54.137498 14812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:54.143307 14812 catalog_manager.cc:1347] Generated new cluster ID: 744c2f624e6e41a7b335fd5a75126e4a
I20260621 07:51:54.143388 14812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:54.171185 14812 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:54.172197 14812 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:54.186899 14812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b7c0bd52e224279a80fe020bce3b23e: Generated new TSK 0
I20260621 07:51:54.187883 14812 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:51:54.218461 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:36171
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:54.328922 14829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:54.329228 14829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:54.329306 14829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:54.333719 14829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:54.333796 14829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:54.333891 14829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:51:54.339203 14829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:36171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:54.340238 14829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:54.341234 14829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:54.350744 14834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:54.350829 14835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:54.350764 14829 server_base.cc:1034] running on GCE node
W20260621 07:51:54.350786 14837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:54.351945 14829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:54.352555 14829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:54.353749 14829 hybrid_clock.cc:648] HybridClock initialized: now 1782028314353735 us; error 30 us; skew 500 ppm
Jun 21 07:51:54 dist-test-slave-lbpt krb5kdc[14722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028314, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:54.356491 14829 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:51:54.357327 14829 webserver.cc:460] Webserver started at http://127.12.12.1:33171/ using document root <none> and password file <none>
I20260621 07:51:54.357834 14829 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:54.357892 14829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:54.358047 14829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:54.360011 14829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "2edca8a9a9564226afec55511532e954"
format_stamp: "Formatted at 2026-06-21 07:51:54 on dist-test-slave-lbpt"
server_key: "1c21203dbd796faf6fc26733ba97b866"
server_key_iv: "bf9e52e83a5d40da69da71d3b552405f"
server_key_version: "encryptionkey@0"
I20260621 07:51:54.360495 14829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "2edca8a9a9564226afec55511532e954"
format_stamp: "Formatted at 2026-06-21 07:51:54 on dist-test-slave-lbpt"
server_key: "1c21203dbd796faf6fc26733ba97b866"
server_key_iv: "bf9e52e83a5d40da69da71d3b552405f"
server_key_version: "encryptionkey@0"
I20260621 07:51:54.364192 14829 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260621 07:51:54.366611 14844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:54.368023 14829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:51:54.368156 14829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "2edca8a9a9564226afec55511532e954"
format_stamp: "Formatted at 2026-06-21 07:51:54 on dist-test-slave-lbpt"
server_key: "1c21203dbd796faf6fc26733ba97b866"
server_key_iv: "bf9e52e83a5d40da69da71d3b552405f"
server_key_version: "encryptionkey@0"
I20260621 07:51:54.368276 14829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:54.389055 14829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:54.392700 14829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:54.392927 14829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:54.393576 14829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:54.394543 14829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:54.394621 14829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:54.394690 14829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:54.394740 14829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:54.404224 14829 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:43523
I20260621 07:51:54.404255 14957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:43523 every 8 connection(s)
I20260621 07:51:54.405166 14829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:51:54.405414 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 14829
I20260621 07:51:54.405539 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:51:54.405853 12336 external_mini_cluster.cc:1428] Setting key 360b0a179753458545e84d1990bd924c
Jun 21 07:51:54 dist-test-slave-lbpt krb5kdc[14722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028314, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
I20260621 07:51:54.420899 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:54.407220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:59173 (local address 127.12.12.62:36171)
0621 07:51:54.407359 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:54.407364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:54.408368 (+ 1004us) server_negotiation.cc:407] Connection header received
0621 07:51:54.409647 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:54.409650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:54.409703 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:54.409788 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:54.411243 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:54.411800 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:54.413232 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:54.413469 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:54.413600 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:54.416634 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:54.416637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:54.416639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:54.416663 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:54.418491 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:54.419128 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:54.419131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:54.419132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:54.419213 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:54.419553 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:54.419557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:54.419558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:54.419877 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:54.420012 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:54.420629 (+ 617us) server_negotiation.cc:299] Negotiation successful
0621 07:51:54.420767 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260621 07:51:54.422031 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:54.407720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:36171 (local address 127.12.12.1:59173)
0621 07:51:54.408223 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:54.408266 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:54.409446 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:54.409975 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:54.409982 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:54.410339 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:54.411073 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:54.411096 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:54.411993 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:54.412000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:54.413114 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:54.413123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:54.413681 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:54.413686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:54.413930 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:54.414697 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:54.414721 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:54.416447 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:54.418661 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:54.418666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:54.418679 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:54.419009 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:54.419297 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:54.419299 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:54.419301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:54.419436 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:54.420002 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:54.420007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:54.420350 (+ 343us) client_negotiation.cc:769] Sending connection context
0621 07:51:54.420574 (+ 224us) client_negotiation.cc:240] Negotiation successful
0621 07:51:54.420971 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":142,"threads_started":1}
I20260621 07:51:54.423480 14958 heartbeater.cc:347] Connected to a master server at 127.12.12.62:36171
I20260621 07:51:54.423758 14958 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:54.424309 14958 heartbeater.cc:511] Master 127.12.12.62:36171 requested a full tablet report, sending...
I20260621 07:51:54.426124 14767 ts_manager.cc:194] Registered new tserver with Master: 2edca8a9a9564226afec55511532e954 (127.12.12.1:43523)
I20260621 07:51:54.427865 14767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:59173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:51:54.437378 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:36171
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:54.552488 14963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:54.552804 14963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:54.552878 14963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:54.557019 14963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:54.557116 14963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:54.557205 14963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:51:54.562831 14963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:36171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:54.563961 14963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:54.565061 14963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:54.575142 14968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:54.575151 14971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:54.575291 14963 server_base.cc:1034] running on GCE node
W20260621 07:51:54.575166 14969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:54.575876 14963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:54.576599 14963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:54.577791 14963 hybrid_clock.cc:648] HybridClock initialized: now 1782028314577758 us; error 53 us; skew 500 ppm
Jun 21 07:51:54 dist-test-slave-lbpt krb5kdc[14722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028314, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:54.580667 14963 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:51:54.581513 14963 webserver.cc:460] Webserver started at http://127.12.12.2:42613/ using document root <none> and password file <none>
I20260621 07:51:54.582044 14963 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:54.582103 14963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:54.582268 14963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:54.584360 14963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "cc1241571fd643c98ad1c0d460036e21"
format_stamp: "Formatted at 2026-06-21 07:51:54 on dist-test-slave-lbpt"
server_key: "b886036b3fda328572e799401e72684c"
server_key_iv: "7c1486571482ff29587f0143575ff3f8"
server_key_version: "encryptionkey@0"
I20260621 07:51:54.584870 14963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "cc1241571fd643c98ad1c0d460036e21"
format_stamp: "Formatted at 2026-06-21 07:51:54 on dist-test-slave-lbpt"
server_key: "b886036b3fda328572e799401e72684c"
server_key_iv: "7c1486571482ff29587f0143575ff3f8"
server_key_version: "encryptionkey@0"
I20260621 07:51:54.588573 14963 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260621 07:51:54.591073 14978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:54.592338 14963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:51:54.592455 14963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "cc1241571fd643c98ad1c0d460036e21"
format_stamp: "Formatted at 2026-06-21 07:51:54 on dist-test-slave-lbpt"
server_key: "b886036b3fda328572e799401e72684c"
server_key_iv: "7c1486571482ff29587f0143575ff3f8"
server_key_version: "encryptionkey@0"
I20260621 07:51:54.592566 14963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:54.630000 14963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:54.633399 14963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:54.633556 14963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:54.634298 14963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:54.635212 14963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:54.635259 14963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:54.635324 14963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:54.635355 14963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:54.644479 14963 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:37485
I20260621 07:51:54.644518 15091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:37485 every 8 connection(s)
I20260621 07:51:54.645417 14963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:51:54.654541 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 14963
I20260621 07:51:54.654693 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
Jun 21 07:51:54 dist-test-slave-lbpt krb5kdc[14722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028314, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:54.654986 12336 external_mini_cluster.cc:1428] Setting key 92ac294115f018af58cdb36a34584266
I20260621 07:51:54.659669 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:54.647290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:41203 (local address 127.12.12.62:36171)
0621 07:51:54.647425 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:54.647429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:54.648230 (+ 801us) server_negotiation.cc:407] Connection header received
0621 07:51:54.649516 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:54.649520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:54.649576 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:54.649690 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:54.650917 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:54.651447 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:54.652592 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:54.652830 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:54.652945 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:54.655647 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:54.655651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:54.655654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:54.655678 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:54.657227 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:54.657871 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:54.657874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:54.657875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:54.657949 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:54.658370 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:54.658375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:54.658378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:54.658671 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:54.658782 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:54.659280 (+ 498us) server_negotiation.cc:299] Negotiation successful
0621 07:51:54.659425 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:51:54.660429 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:54.647636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:36171 (local address 127.12.12.2:41203)
0621 07:51:54.648072 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:54.648115 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:54.649309 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:54.649863 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:54.649870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:54.650227 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:54.650774 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:54.650787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:54.651572 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:54.651577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:54.652461 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:54.652470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:54.652983 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:54.652987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:54.653163 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:54.653703 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:54.653718 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:54.655437 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:54.657347 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:54.657356 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:54.657376 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:54.657771 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:54.658036 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:54.658042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:54.658047 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:54.658257 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:54.658781 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:54.658787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:54.659031 (+ 244us) client_negotiation.cc:769] Sending connection context
0621 07:51:54.659197 (+ 166us) client_negotiation.cc:240] Negotiation successful
0621 07:51:54.659465 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
I20260621 07:51:54.661768 15092 heartbeater.cc:347] Connected to a master server at 127.12.12.62:36171
I20260621 07:51:54.662011 15092 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:54.662596 15092 heartbeater.cc:511] Master 127.12.12.62:36171 requested a full tablet report, sending...
I20260621 07:51:54.663698 14767 ts_manager.cc:194] Registered new tserver with Master: cc1241571fd643c98ad1c0d460036e21 (127.12.12.2:37485)
I20260621 07:51:54.664302 14767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:41203
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:51:54.682580 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:36171
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:54.794615 15097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:54.794881 15097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:54.794924 15097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:54.799010 15097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:54.799074 15097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:54.799134 15097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:51:54.804354 15097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:36171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:54.805405 15097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:54.806406 15097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:54.816208 15102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:54.816212 15103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:54.816293 15105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:54.816390 15097 server_base.cc:1034] running on GCE node
I20260621 07:51:54.816941 15097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:54.817555 15097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:54.818742 15097 hybrid_clock.cc:648] HybridClock initialized: now 1782028314818693 us; error 60 us; skew 500 ppm
Jun 21 07:51:54 dist-test-slave-lbpt krb5kdc[14722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028314, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:54.821660 15097 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:51:54.822713 15097 webserver.cc:460] Webserver started at http://127.12.12.3:43037/ using document root <none> and password file <none>
I20260621 07:51:54.823319 15097 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:54.823414 15097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:54.823664 15097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:54.825781 15097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "c10c6843d4c545b0831d91ee880cc0e2"
format_stamp: "Formatted at 2026-06-21 07:51:54 on dist-test-slave-lbpt"
server_key: "b498a60bf4c15ce4201ba0d177e60231"
server_key_iv: "e9554930bc93b941cbe65b0c76399f5f"
server_key_version: "encryptionkey@0"
I20260621 07:51:54.826363 15097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "c10c6843d4c545b0831d91ee880cc0e2"
format_stamp: "Formatted at 2026-06-21 07:51:54 on dist-test-slave-lbpt"
server_key: "b498a60bf4c15ce4201ba0d177e60231"
server_key_iv: "e9554930bc93b941cbe65b0c76399f5f"
server_key_version: "encryptionkey@0"
I20260621 07:51:54.830253 15097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 07:51:54.832826 15112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:54.834121 15097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:54.834251 15097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "c10c6843d4c545b0831d91ee880cc0e2"
format_stamp: "Formatted at 2026-06-21 07:51:54 on dist-test-slave-lbpt"
server_key: "b498a60bf4c15ce4201ba0d177e60231"
server_key_iv: "e9554930bc93b941cbe65b0c76399f5f"
server_key_version: "encryptionkey@0"
I20260621 07:51:54.834374 15097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:54.844732 15097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:54.847894 15097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:54.848115 15097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:54.848762 15097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:54.849611 15097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:54.849684 15097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:54.849756 15097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:54.849797 15097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:54.858964 15097 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:39335
I20260621 07:51:54.858975 15225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:39335 every 8 connection(s)
I20260621 07:51:54.859882 15097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:51:54.868911 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 15097
I20260621 07:51:54.869045 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:51:54.869354 12336 external_mini_cluster.cc:1428] Setting key 9eb28c21deeb76ce0a318afb5dcc281b
Jun 21 07:51:54 dist-test-slave-lbpt krb5kdc[14722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028314, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:54.873888 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:54.861718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:55929 (local address 127.12.12.62:36171)
0621 07:51:54.861860 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:54.861864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:54.862881 (+ 1017us) server_negotiation.cc:407] Connection header received
0621 07:51:54.864209 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:54.864213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:54.864273 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:54.864382 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:54.865580 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:54.866109 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:54.867321 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:54.867549 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:54.867661 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:54.870266 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:54.870269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:54.870271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:54.870296 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:54.871889 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:54.872405 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:54.872410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:54.872413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:54.872492 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:54.872831 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:54.872834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:54.872836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:54.873014 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:54.873123 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:54.873601 (+ 478us) server_negotiation.cc:299] Negotiation successful
0621 07:51:54.873728 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:51:54.874758 15228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:54.862145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:36171 (local address 127.12.12.3:55929)
0621 07:51:54.862724 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:54.862769 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:54.863972 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:54.864561 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:54.864569 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:54.864936 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:54.865407 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:54.865420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:54.866244 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:54.866247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:54.867194 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:54.867203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:54.867701 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:54.867706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:54.867873 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:54.868432 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:54.868448 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:54.870088 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:54.872029 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:54.872035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:54.872049 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:54.872304 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:54.872590 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:54.872592 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:54.872594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:54.872731 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:54.873127 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:54.873136 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:54.873377 (+ 241us) client_negotiation.cc:769] Sending connection context
0621 07:51:54.873564 (+ 187us) client_negotiation.cc:240] Negotiation successful
0621 07:51:54.873906 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":105,"threads_started":1}
I20260621 07:51:54.875829 15226 heartbeater.cc:347] Connected to a master server at 127.12.12.62:36171
I20260621 07:51:54.876108 15226 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:54.876653 15226 heartbeater.cc:511] Master 127.12.12.62:36171 requested a full tablet report, sending...
I20260621 07:51:54.877715 14767 ts_manager.cc:194] Registered new tserver with Master: c10c6843d4c545b0831d91ee880cc0e2 (127.12.12.3:39335)
I20260621 07:51:54.878252 14767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:55929
I20260621 07:51:54.883571 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 07:51:54.897368 14806 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:54.895751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.12.12.62:36171)
0621 07:51:54.895909 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:54.895915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:54.896052 (+ 137us) server_negotiation.cc:407] Connection header received
0621 07:51:54.896253 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:54.896257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:54.896311 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:54.896397 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:54.897253 (+ 856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44030: BlockingRecv error: recv got EOF from 127.0.0.1:44030 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:51:54.898492 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 14829
I20260621 07:51:54.904233 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 14963
I20260621 07:51:54.909662 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 15097
I20260621 07:51:54.914896 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 14735
2026-06-21T07:51:54Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2161 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:54 dist-test-slave-lbpt krb5kdc[15238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:54 dist-test-slave-lbpt krb5kdc[15238](info): set up 2 sockets
Jun 21 07:51:54 dist-test-slave-lbpt krb5kdc[15238](info): commencing operation
krb5kdc: starting...
W20260621 07:51:55.954487 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:51:56 dist-test-slave-lbpt krb5kdc[15238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:51:56.083540 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:43175
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:43329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:43175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:56.196722 15251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:56.197024 15251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:56.197065 15251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:56.201210 15251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:56.201287 15251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:56.201308 15251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:56.201324 15251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:56.201340 15251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:56.206997 15251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:43329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:43175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:43175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:56.208214 15251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:56.209543 15251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:56.218570 15256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:56.218516 15259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:56.218624 15257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:56.218732 15251 server_base.cc:1034] running on GCE node
I20260621 07:51:56.219393 15251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:56.220563 15251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:56.221742 15251 hybrid_clock.cc:648] HybridClock initialized: now 1782028316221703 us; error 55 us; skew 500 ppm
Jun 21 07:51:56 dist-test-slave-lbpt krb5kdc[15238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028316, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:56.227917 15251 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:51:56.228956 15251 webserver.cc:460] Webserver started at http://127.12.12.62:36103/ using document root <none> and password file <none>
I20260621 07:51:56.229574 15251 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:56.229640 15251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:56.229864 15251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:56.232070 15251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "977eaf9520674f78815a977e26c5f5ca"
format_stamp: "Formatted at 2026-06-21 07:51:56 on dist-test-slave-lbpt"
server_key: "28f4f4b120b84511b4d412da3c8507e9"
server_key_iv: "8a75ee94c8577b43b31bdb3b00e261a2"
server_key_version: "encryptionkey@0"
I20260621 07:51:56.232641 15251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "977eaf9520674f78815a977e26c5f5ca"
format_stamp: "Formatted at 2026-06-21 07:51:56 on dist-test-slave-lbpt"
server_key: "28f4f4b120b84511b4d412da3c8507e9"
server_key_iv: "8a75ee94c8577b43b31bdb3b00e261a2"
server_key_version: "encryptionkey@0"
I20260621 07:51:56.236493 15251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:51:56.239005 15266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:56.240506 15251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:51:56.240654 15251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "977eaf9520674f78815a977e26c5f5ca"
format_stamp: "Formatted at 2026-06-21 07:51:56 on dist-test-slave-lbpt"
server_key: "28f4f4b120b84511b4d412da3c8507e9"
server_key_iv: "8a75ee94c8577b43b31bdb3b00e261a2"
server_key_version: "encryptionkey@0"
I20260621 07:51:56.240765 15251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:56.255342 15251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:56.258015 15251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:56.258224 15251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:56.265300 15251 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:43175
I20260621 07:51:56.265668 15318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:43175 every 8 connection(s)
I20260621 07:51:56.266336 15251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:56.269112 15319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:56.270057 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 15251
I20260621 07:51:56.270146 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:51:56.270483 12336 external_mini_cluster.cc:1428] Setting key 02dede9b0a926f3b9efe38f016af2dc3
I20260621 07:51:56.275441 15319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca: Bootstrap starting.
I20260621 07:51:56.278088 15319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:56.279031 15319 log.cc:824] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:51:56 dist-test-slave-lbpt krb5kdc[15238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:56.281165 15319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca: No bootstrap required, opened a new log
I20260621 07:51:56.284662 15319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977eaf9520674f78815a977e26c5f5ca" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43175 } }
I20260621 07:51:56.284976 15319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:56.285056 15319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 977eaf9520674f78815a977e26c5f5ca, State: Initialized, Role: FOLLOWER
I20260621 07:51:56.285514 15319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977eaf9520674f78815a977e26c5f5ca" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43175 } }
I20260621 07:51:56.285665 15319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:56.285753 15319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:56.285764 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:56.271989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.12.12.62:43175)
0621 07:51:56.272439 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:56.272455 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:56.272491 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:51:56.273132 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:56.273149 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:56.273494 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:56.273780 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:56.274807 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:56.275945 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:56.277174 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:56.277703 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:56.277826 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:56.280938 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:56.280944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:56.280957 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:56.280984 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:56.282772 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:56.283242 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:56.283247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:56.283252 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:56.283359 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:56.283649 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:56.283654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:56.283656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:56.284071 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:56.284287 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:56.284521 (+ 234us) server_negotiation.cc:299] Negotiation successful
0621 07:51:56.284813 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":146,"threads_started":1}
I20260621 07:51:56.285899 15319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:56.286933 15319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977eaf9520674f78815a977e26c5f5ca" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43175 } }
I20260621 07:51:56.287288 15319 leader_election.cc:302] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 977eaf9520674f78815a977e26c5f5ca; no voters:
I20260621 07:51:56.287665 15319 leader_election.cc:288] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:56.287742 15324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:56.288024 15324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [term 1 LEADER]: Becoming Leader. State: Replica: 977eaf9520674f78815a977e26c5f5ca, State: Running, Role: LEADER
I20260621 07:51:56.288401 15324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977eaf9520674f78815a977e26c5f5ca" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43175 } }
I20260621 07:51:56.288791 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:56.289088 15319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:56.289664 15324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "977eaf9520674f78815a977e26c5f5ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977eaf9520674f78815a977e26c5f5ca" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43175 } } }
I20260621 07:51:56.289873 15324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:56.290208 15325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 977eaf9520674f78815a977e26c5f5ca. Latest consensus state: current_term: 1 leader_uuid: "977eaf9520674f78815a977e26c5f5ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977eaf9520674f78815a977e26c5f5ca" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43175 } } }
I20260621 07:51:56.290335 15325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca [sys.catalog]: This master's current role is: LEADER
W20260621 07:51:56.294143 15340 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:51:56.294243 15340 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:51:56.294397 15331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:56.295401 15331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:56.301566 15331 catalog_manager.cc:1347] Generated new cluster ID: a01219005e964f879e1efe8a09df20b8
I20260621 07:51:56.301646 15331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:56.311535 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:56.306490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51734 (local address 127.12.12.62:43175)
0621 07:51:56.306624 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:56.306629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:56.306822 (+ 193us) server_negotiation.cc:407] Connection header received
0621 07:51:56.306986 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:56.306989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:56.307042 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:56.307133 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:56.307679 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:56.308249 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:56.309121 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:56.309362 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:56.309471 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:56.310459 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:56.310463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:56.310465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:56.311034 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:56.311070 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:56.311092 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:56.311206 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:56.311316 (+ 110us) server_negotiation.cc:299] Negotiation successful
0621 07:51:56.311380 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 07:51:56.311635 15331 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:56.312839 15331 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:56.334949 15331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 977eaf9520674f78815a977e26c5f5ca: Generated new TSK 0
I20260621 07:51:56.335892 15331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:56.341115 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 15251
2026-06-21T07:51:56Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1424 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:56 dist-test-slave-lbpt krb5kdc[15352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:56 dist-test-slave-lbpt krb5kdc[15352](info): set up 2 sockets
Jun 21 07:51:56 dist-test-slave-lbpt krb5kdc[15352](info): commencing operation
krb5kdc: starting...
W20260621 07:51:57.378355 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:51:57 dist-test-slave-lbpt krb5kdc[15352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:51:57.497493 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:42331
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:42331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:57.612632 15365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:57.612977 15365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:57.613061 15365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:57.617204 15365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:57.617308 15365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:57.617343 15365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:57.617384 15365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:57.617432 15365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:57.622949 15365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:42331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:42331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:57.624049 15365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:57.625082 15365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:57.633615 15370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:57.633623 15371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:57.633623 15373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:57.633821 15365 server_base.cc:1034] running on GCE node
I20260621 07:51:57.634580 15365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:57.635670 15365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:57.636834 15365 hybrid_clock.cc:648] HybridClock initialized: now 1782028317636810 us; error 38 us; skew 500 ppm
Jun 21 07:51:57 dist-test-slave-lbpt krb5kdc[15352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:57.639740 15365 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:51:57.640643 15365 webserver.cc:460] Webserver started at http://127.12.12.62:39689/ using document root <none> and password file <none>
I20260621 07:51:57.641168 15365 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:57.641223 15365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:57.641384 15365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:57.643424 15365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "071667f9462646c8aa9d87c5ffa27738"
format_stamp: "Formatted at 2026-06-21 07:51:57 on dist-test-slave-lbpt"
server_key: "7541f4bde03d758f69516913e5a2c8ec"
server_key_iv: "fb1b7ccd2435a11f2833c64889559448"
server_key_version: "encryptionkey@0"
I20260621 07:51:57.643924 15365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "071667f9462646c8aa9d87c5ffa27738"
format_stamp: "Formatted at 2026-06-21 07:51:57 on dist-test-slave-lbpt"
server_key: "7541f4bde03d758f69516913e5a2c8ec"
server_key_iv: "fb1b7ccd2435a11f2833c64889559448"
server_key_version: "encryptionkey@0"
I20260621 07:51:57.647514 15365 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260621 07:51:57.649807 15380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:57.651206 15365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 07:51:57.651345 15365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "071667f9462646c8aa9d87c5ffa27738"
format_stamp: "Formatted at 2026-06-21 07:51:57 on dist-test-slave-lbpt"
server_key: "7541f4bde03d758f69516913e5a2c8ec"
server_key_iv: "fb1b7ccd2435a11f2833c64889559448"
server_key_version: "encryptionkey@0"
I20260621 07:51:57.651458 15365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:57.681819 15365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:57.685325 15365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:57.685545 15365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:57.692638 15365 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:42331
I20260621 07:51:57.692930 15432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:42331 every 8 connection(s)
I20260621 07:51:57.693658 15365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:57.694132 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 15365
I20260621 07:51:57.694280 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:51:57.694660 12336 external_mini_cluster.cc:1428] Setting key 5f6bde97ca175fa5437b4339cf88e2c6
I20260621 07:51:57.696942 15433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:57.702844 15433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738: Bootstrap starting.
Jun 21 07:51:57 dist-test-slave-lbpt krb5kdc[15352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:57.705384 15433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:57.706144 15433 log.cc:824] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:57.708424 15433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738: No bootstrap required, opened a new log
I20260621 07:51:57.709622 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:57.696070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60790 (local address 127.12.12.62:42331)
0621 07:51:57.696526 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:57.696542 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:57.696580 (+ 38us) server_negotiation.cc:407] Connection header received
0621 07:51:57.697330 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:57.697350 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:57.697697 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:57.698065 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:57.698937 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:57.699936 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:57.700870 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:57.701375 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:57.701559 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:57.704226 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:57.704231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:57.704245 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:57.704275 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:57.706514 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:57.707064 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:57.707069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:57.707074 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:57.707188 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:57.707540 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:57.707543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:57.707544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:57.707947 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:57.708125 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:57.708433 (+ 308us) server_negotiation.cc:299] Negotiation successful
0621 07:51:57.708889 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":135,"threads_started":1}
I20260621 07:51:57.712512 15433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071667f9462646c8aa9d87c5ffa27738" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42331 } }
I20260621 07:51:57.712726 15433 raft_consensus.cc:374] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:57.712801 15433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 071667f9462646c8aa9d87c5ffa27738, State: Initialized, Role: FOLLOWER
I20260621 07:51:57.713258 15433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071667f9462646c8aa9d87c5ffa27738" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42331 } }
I20260621 07:51:57.713395 15433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:57.713480 15433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:57.713605 15433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:57.714654 15433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071667f9462646c8aa9d87c5ffa27738" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42331 } }
I20260621 07:51:57.715018 15433 leader_election.cc:302] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 071667f9462646c8aa9d87c5ffa27738; no voters:
I20260621 07:51:57.715358 15433 leader_election.cc:288] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:57.715524 15438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:57.715770 15438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [term 1 LEADER]: Becoming Leader. State: Replica: 071667f9462646c8aa9d87c5ffa27738, State: Running, Role: LEADER
I20260621 07:51:57.716096 15438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071667f9462646c8aa9d87c5ffa27738" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42331 } }
I20260621 07:51:57.716575 15433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:57.717760 15439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "071667f9462646c8aa9d87c5ffa27738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071667f9462646c8aa9d87c5ffa27738" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42331 } } }
I20260621 07:51:57.717890 15439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:57.718266 15447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:57.718246 15440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 071667f9462646c8aa9d87c5ffa27738. Latest consensus state: current_term: 1 leader_uuid: "071667f9462646c8aa9d87c5ffa27738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071667f9462646c8aa9d87c5ffa27738" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42331 } } }
I20260621 07:51:57.718381 15440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:57.721246 15447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:57.726723 15447 catalog_manager.cc:1347] Generated new cluster ID: c56d746ca99d46c88d690d94ac95c0c7
I20260621 07:51:57.726805 15447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:57.737834 15447 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:57.739080 15447 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:57.749614 15447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 071667f9462646c8aa9d87c5ffa27738: Generated new TSK 0
I20260621 07:51:57.750545 15447 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:51:57.784649 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:42331
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:57.898325 15459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:57.898644 15459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:57.898689 15459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:57.902925 15459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:57.902999 15459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:57.903092 15459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:51:57.908565 15459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:42331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:57.909777 15459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:57.911073 15459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:57.921288 15465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:57.921314 15467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:57.921481 15459 server_base.cc:1034] running on GCE node
W20260621 07:51:57.921314 15464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:57.922070 15459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:57.922905 15459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:57.924135 15459 hybrid_clock.cc:648] HybridClock initialized: now 1782028317924078 us; error 70 us; skew 500 ppm
Jun 21 07:51:57 dist-test-slave-lbpt krb5kdc[15352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:57.927260 15459 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:51:57.928261 15459 webserver.cc:460] Webserver started at http://127.12.12.1:36329/ using document root <none> and password file <none>
I20260621 07:51:57.928901 15459 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:57.929008 15459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:57.929251 15459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:57.931499 15459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "a6c26250835b40dabe9e8966ec755aae"
format_stamp: "Formatted at 2026-06-21 07:51:57 on dist-test-slave-lbpt"
server_key: "2171418b8c85286debfe702cff9b9330"
server_key_iv: "dc0e068d3e117822be1ff8c7edef372f"
server_key_version: "encryptionkey@0"
I20260621 07:51:57.932080 15459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "a6c26250835b40dabe9e8966ec755aae"
format_stamp: "Formatted at 2026-06-21 07:51:57 on dist-test-slave-lbpt"
server_key: "2171418b8c85286debfe702cff9b9330"
server_key_iv: "dc0e068d3e117822be1ff8c7edef372f"
server_key_version: "encryptionkey@0"
I20260621 07:51:57.935938 15459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:51:57.938859 15474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:57.940232 15459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:51:57.940402 15459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "a6c26250835b40dabe9e8966ec755aae"
format_stamp: "Formatted at 2026-06-21 07:51:57 on dist-test-slave-lbpt"
server_key: "2171418b8c85286debfe702cff9b9330"
server_key_iv: "dc0e068d3e117822be1ff8c7edef372f"
server_key_version: "encryptionkey@0"
I20260621 07:51:57.940574 15459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:57.968453 15459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:57.971753 15459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:57.971958 15459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:57.972563 15459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:57.973448 15459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:57.973536 15459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:57.973634 15459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:57.973688 15459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:57.982939 15459 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:41783
I20260621 07:51:57.982962 15587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:41783 every 8 connection(s)
I20260621 07:51:57.983883 15459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:51:57.991673 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 15459
I20260621 07:51:57.991847 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:51:57.992203 12336 external_mini_cluster.cc:1428] Setting key 0b5b6ba1a6af0247c1d45a06d5b1b91a
Jun 21 07:51:57 dist-test-slave-lbpt krb5kdc[15352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:57.999691 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:57.985743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:49599 (local address 127.12.12.62:42331)
0621 07:51:57.985917 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:57.985922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:57.986840 (+ 918us) server_negotiation.cc:407] Connection header received
0621 07:51:57.988173 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:57.988176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:57.988223 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:57.988290 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:57.989458 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:57.990034 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:57.991181 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:57.991460 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:57.991582 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:57.995056 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:57.995063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:57.995069 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:57.995107 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:57.996829 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:57.997620 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:57.997624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:57.997626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:57.997729 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:57.998210 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:57.998216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:57.998219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:57.998483 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:57.998596 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:57.999342 (+ 746us) server_negotiation.cc:299] Negotiation successful
0621 07:51:57.999496 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:51:58.000921 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:57.986174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:42331 (local address 127.12.12.1:49599)
0621 07:51:57.986690 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:57.986734 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:57.987952 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:57.988456 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:57.988463 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:57.988824 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:57.989317 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:57.989331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:57.990154 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:57.990157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:57.991067 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:57.991076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:57.991615 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:57.991618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:57.991755 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:57.992510 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:57.992535 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:57.994752 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:57.996970 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:57.996977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:57.996996 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:57.997488 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:57.997842 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:57.997845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:57.997848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:57.998076 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:57.998608 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:57.998619 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:57.998990 (+ 371us) client_negotiation.cc:769] Sending connection context
0621 07:51:57.999257 (+ 267us) client_negotiation.cc:240] Negotiation successful
0621 07:51:57.999665 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":109,"threads_started":1}
I20260621 07:51:58.002560 15588 heartbeater.cc:347] Connected to a master server at 127.12.12.62:42331
I20260621 07:51:58.002861 15588 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:58.003420 15588 heartbeater.cc:511] Master 127.12.12.62:42331 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
I20260621 07:51:58.005196 15397 ts_manager.cc:194] Registered new tserver with Master: a6c26250835b40dabe9e8966ec755aae (127.12.12.1:41783)
I20260621 07:51:58.006608 15397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:49599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:51:58.023985 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:42331
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:58.136137 15593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:58.136396 15593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:58.136441 15593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:58.140507 15593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:58.140559 15593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:58.140653 15593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:51:58.145987 15593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:42331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:58.147092 15593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:58.148138 15593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:58.158056 15598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:58.158083 15601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:58.158252 15599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:58.158278 15593 server_base.cc:1034] running on GCE node
I20260621 07:51:58.158733 15593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:58.159368 15593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:58.160545 15593 hybrid_clock.cc:648] HybridClock initialized: now 1782028318160517 us; error 52 us; skew 500 ppm
Jun 21 07:51:58 dist-test-slave-lbpt krb5kdc[15352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028318, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:58.163849 15593 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:51:58.164709 15593 webserver.cc:460] Webserver started at http://127.12.12.2:40867/ using document root <none> and password file <none>
I20260621 07:51:58.165270 15593 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:58.165331 15593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:58.165541 15593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:58.167666 15593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "69c2ce9f4f44408fafa2d263bef8ae6b"
format_stamp: "Formatted at 2026-06-21 07:51:58 on dist-test-slave-lbpt"
server_key: "86c73feb28974928840c53aa180eb3f2"
server_key_iv: "3f7a1dce35c276d2a2674a0f66910c84"
server_key_version: "encryptionkey@0"
I20260621 07:51:58.168270 15593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "69c2ce9f4f44408fafa2d263bef8ae6b"
format_stamp: "Formatted at 2026-06-21 07:51:58 on dist-test-slave-lbpt"
server_key: "86c73feb28974928840c53aa180eb3f2"
server_key_iv: "3f7a1dce35c276d2a2674a0f66910c84"
server_key_version: "encryptionkey@0"
I20260621 07:51:58.172228 15593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:51:58.174693 15608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:58.175957 15593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:51:58.176090 15593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "69c2ce9f4f44408fafa2d263bef8ae6b"
format_stamp: "Formatted at 2026-06-21 07:51:58 on dist-test-slave-lbpt"
server_key: "86c73feb28974928840c53aa180eb3f2"
server_key_iv: "3f7a1dce35c276d2a2674a0f66910c84"
server_key_version: "encryptionkey@0"
I20260621 07:51:58.176206 15593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:58.189808 15593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:58.193292 15593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:58.193518 15593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:58.194154 15593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:58.195053 15593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:58.195137 15593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:58.195212 15593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:58.195266 15593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:58.204659 15593 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:36347
I20260621 07:51:58.204676 15721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:36347 every 8 connection(s)
I20260621 07:51:58.205634 15593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:51:58.210363 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 15593
I20260621 07:51:58.210543 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:51:58.210811 12336 external_mini_cluster.cc:1428] Setting key aced15c102bd6302ae267980322499d8
Jun 21 07:51:58 dist-test-slave-lbpt krb5kdc[15352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028318, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
I20260621 07:51:58.221271 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:58.207553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:53711 (local address 127.12.12.62:42331)
0621 07:51:58.207723 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:58.207728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:58.208526 (+ 798us) server_negotiation.cc:407] Connection header received
0621 07:51:58.209868 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:58.209871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:58.209920 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:58.210009 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:58.211455 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:58.212060 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:58.213821 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:58.214096 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:58.214229 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:58.216962 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:58.216965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:58.216967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:58.216992 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:58.218919 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:58.219513 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:58.219517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:58.219520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:58.219665 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:58.220026 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:58.220031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:58.220034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:58.220301 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:58.220422 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:58.220973 (+ 551us) server_negotiation.cc:299] Negotiation successful
0621 07:51:58.221115 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 07:51:58.222091 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:58.207927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:42331 (local address 127.12.12.2:53711)
0621 07:51:58.208371 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:58.208415 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:58.209639 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:58.210188 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:58.210196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:58.210707 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:58.211245 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:58.211260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:58.212210 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:58.212213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:58.213665 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:58.213680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:58.214270 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:58.214274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:58.214523 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:58.215216 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:58.215233 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:58.216773 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:58.219060 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:58.219065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:58.219078 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:58.219357 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:58.219770 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:58.219773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:58.219774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:58.219922 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:58.220444 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:58.220449 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:58.220693 (+ 244us) client_negotiation.cc:769] Sending connection context
0621 07:51:58.220912 (+ 219us) client_negotiation.cc:240] Negotiation successful
0621 07:51:58.221205 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
I20260621 07:51:58.223569 15722 heartbeater.cc:347] Connected to a master server at 127.12.12.62:42331
I20260621 07:51:58.223843 15722 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:58.224493 15722 heartbeater.cc:511] Master 127.12.12.62:42331 requested a full tablet report, sending...
I20260621 07:51:58.225862 15397 ts_manager.cc:194] Registered new tserver with Master: 69c2ce9f4f44408fafa2d263bef8ae6b (127.12.12.2:36347)
I20260621 07:51:58.226522 15397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:53711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:51:58.237766 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:42331
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:58.351485 15727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:58.351841 15727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:58.351918 15727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:58.356658 15727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:58.356750 15727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:58.356840 15727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:51:58.362387 15727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:42331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:58.363641 15727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:58.364681 15727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:58.374676 15733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:58.374676 15732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:58.374688 15735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:58.374822 15727 server_base.cc:1034] running on GCE node
I20260621 07:51:58.375396 15727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:58.376060 15727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:58.377259 15727 hybrid_clock.cc:648] HybridClock initialized: now 1782028318377225 us; error 49 us; skew 500 ppm
Jun 21 07:51:58 dist-test-slave-lbpt krb5kdc[15352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028318, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:58.380142 15727 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:51:58.381165 15727 webserver.cc:460] Webserver started at http://127.12.12.3:34611/ using document root <none> and password file <none>
I20260621 07:51:58.381749 15727 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:58.381841 15727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:58.382076 15727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:58.384284 15727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "1ee8183ec09247ed81b44f43f186e89a"
format_stamp: "Formatted at 2026-06-21 07:51:58 on dist-test-slave-lbpt"
server_key: "7efc5b4c380911d721064fff087fe327"
server_key_iv: "cda748649a7b11fe3f53636df80e3f54"
server_key_version: "encryptionkey@0"
I20260621 07:51:58.384856 15727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "1ee8183ec09247ed81b44f43f186e89a"
format_stamp: "Formatted at 2026-06-21 07:51:58 on dist-test-slave-lbpt"
server_key: "7efc5b4c380911d721064fff087fe327"
server_key_iv: "cda748649a7b11fe3f53636df80e3f54"
server_key_version: "encryptionkey@0"
I20260621 07:51:58.388782 15727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:51:58.391530 15742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:58.392776 15727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 07:51:58.392917 15727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "1ee8183ec09247ed81b44f43f186e89a"
format_stamp: "Formatted at 2026-06-21 07:51:58 on dist-test-slave-lbpt"
server_key: "7efc5b4c380911d721064fff087fe327"
server_key_iv: "cda748649a7b11fe3f53636df80e3f54"
server_key_version: "encryptionkey@0"
I20260621 07:51:58.393036 15727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:58.416316 15727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:58.419960 15727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:58.420173 15727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:58.420780 15727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:58.421622 15727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:58.421691 15727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:58.421756 15727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:58.421808 15727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:58.431283 15727 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:35683
I20260621 07:51:58.431300 15855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:35683 every 8 connection(s)
I20260621 07:51:58.432215 15727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:51:58.434674 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 15727
I20260621 07:51:58.434796 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:51:58.435137 12336 external_mini_cluster.cc:1428] Setting key 54d6716612233bfd0b2c65d52255c90d
Jun 21 07:51:58 dist-test-slave-lbpt krb5kdc[15352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028318, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:58.446513 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:58.434071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:55959 (local address 127.12.12.62:42331)
0621 07:51:58.434218 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:58.434222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:58.435282 (+ 1060us) server_negotiation.cc:407] Connection header received
0621 07:51:58.436986 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:58.436991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:58.437045 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:58.437134 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:58.438347 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:58.439008 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:58.440165 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:58.440391 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:58.440502 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:58.443218 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:58.443222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:58.443224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:58.443251 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:58.444618 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:58.445118 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:58.445120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:58.445121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:58.445195 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:58.445480 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:58.445482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:58.445483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:58.445640 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:58.445710 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:58.446234 (+ 524us) server_negotiation.cc:299] Negotiation successful
0621 07:51:58.446328 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:51:58.447428 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:58.434539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:42331 (local address 127.12.12.3:55959)
0621 07:51:58.435116 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:58.435175 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:58.436783 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:58.437342 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:58.437349 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:58.437725 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:58.438205 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:58.438219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:58.439136 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:58.439140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:58.440040 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:58.440048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:58.440543 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:58.440547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:58.440745 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:58.441365 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:58.441381 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:58.443040 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:58.444724 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:58.444729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:58.444742 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:58.445037 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:58.445272 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:58.445274 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:58.445275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:58.445409 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:58.445748 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:58.445754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:58.446021 (+ 267us) client_negotiation.cc:769] Sending connection context
0621 07:51:58.446190 (+ 169us) client_negotiation.cc:240] Negotiation successful
0621 07:51:58.446545 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":142,"threads_started":1}
I20260621 07:51:58.448700 15856 heartbeater.cc:347] Connected to a master server at 127.12.12.62:42331
I20260621 07:51:58.448985 15856 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:58.449498 15856 heartbeater.cc:511] Master 127.12.12.62:42331 requested a full tablet report, sending...
I20260621 07:51:58.450654 15396 ts_manager.cc:194] Registered new tserver with Master: 1ee8183ec09247ed81b44f43f186e89a (127.12.12.3:35683)
I20260621 07:51:58.451259 15396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:55959
I20260621 07:51:58.460809 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 07:51:58 dist-test-slave-lbpt krb5kdc[15352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028318, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 21 07:51:58 dist-test-slave-lbpt krb5kdc[15352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028318, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:58.487480 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:58.477971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60806 (local address 127.12.12.62:42331)
0621 07:51:58.478152 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:58.478159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:58.478290 (+ 131us) server_negotiation.cc:407] Connection header received
0621 07:51:58.478405 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:58.478448 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:58.478507 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:58.478581 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:58.479388 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:58.479992 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:58.480958 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:58.481184 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:58.481295 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:58.484129 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:58.484133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:58.484135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:58.484164 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:58.485707 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:58.486218 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:58.486220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:58.486221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:58.486297 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:58.486640 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:58.486642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:58.486644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:58.486869 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:58.486975 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:58.487184 (+ 209us) server_negotiation.cc:299] Negotiation successful
0621 07:51:58.487284 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260621 07:51:58.487884 15396 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60806
I20260621 07:51:58.488700 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 15459
I20260621 07:51:58.494259 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 15593
I20260621 07:51:58.499536 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 15727
I20260621 07:51:58.505074 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 15365
2026-06-21T07:51:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2166 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:58 dist-test-slave-lbpt krb5kdc[15868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:58 dist-test-slave-lbpt krb5kdc[15868](info): set up 2 sockets
Jun 21 07:51:58 dist-test-slave-lbpt krb5kdc[15868](info): commencing operation
krb5kdc: starting...
W20260621 07:51:59.562659 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:51:59 dist-test-slave-lbpt krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:51:59.689157 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:34333
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:34333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:59.808061 15881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:59.808439 15881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:59.808532 15881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:59.812780 15881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:59.812892 15881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:59.812929 15881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:59.812974 15881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:59.813009 15881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:59.818714 15881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:34333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:34333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:59.819896 15881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:59.820984 15881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:59.830024 15886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:59.830026 15887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:59.830174 15881 server_base.cc:1034] running on GCE node
W20260621 07:51:59.830025 15889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:59.830948 15881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:59.832155 15881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:59.833364 15881 hybrid_clock.cc:648] HybridClock initialized: now 1782028319833346 us; error 44 us; skew 500 ppm
Jun 21 07:51:59 dist-test-slave-lbpt krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028319, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:59.836491 15881 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:51:59.837617 15881 webserver.cc:460] Webserver started at http://127.12.12.62:40371/ using document root <none> and password file <none>
I20260621 07:51:59.838239 15881 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:59.838335 15881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:59.838613 15881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:59.840713 15881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "e58fb064fb0a46e3b9cf75822c7af414"
format_stamp: "Formatted at 2026-06-21 07:51:59 on dist-test-slave-lbpt"
server_key: "0a8e9ca038a04a5e9d60f3dab52ce1fb"
server_key_iv: "cb6bd2924022648cd428592e50ea0ce1"
server_key_version: "encryptionkey@0"
I20260621 07:51:59.841279 15881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "e58fb064fb0a46e3b9cf75822c7af414"
format_stamp: "Formatted at 2026-06-21 07:51:59 on dist-test-slave-lbpt"
server_key: "0a8e9ca038a04a5e9d60f3dab52ce1fb"
server_key_iv: "cb6bd2924022648cd428592e50ea0ce1"
server_key_version: "encryptionkey@0"
I20260621 07:51:59.845122 15881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 07:51:59.847786 15896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:59.849139 15881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:51:59.849291 15881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "e58fb064fb0a46e3b9cf75822c7af414"
format_stamp: "Formatted at 2026-06-21 07:51:59 on dist-test-slave-lbpt"
server_key: "0a8e9ca038a04a5e9d60f3dab52ce1fb"
server_key_iv: "cb6bd2924022648cd428592e50ea0ce1"
server_key_version: "encryptionkey@0"
I20260621 07:51:59.849412 15881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:59.864591 15881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:59.868141 15881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:59.868362 15881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:59.876092 15881 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:34333
I20260621 07:51:59.876156 15948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:34333 every 8 connection(s)
I20260621 07:51:59.877276 15881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:59.880333 15949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:59.886287 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 15881
I20260621 07:51:59.886446 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:51:59.886739 12336 external_mini_cluster.cc:1428] Setting key 20a4b68a128a6074b74ad9f09f06cbd1
I20260621 07:51:59.886713 15949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414: Bootstrap starting.
I20260621 07:51:59.889424 15949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:59.890314 15949 log.cc:824] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:59.892509 15949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414: No bootstrap required, opened a new log
Jun 21 07:51:59 dist-test-slave-lbpt krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:51:59.897259 15949 raft_consensus.cc:348] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58fb064fb0a46e3b9cf75822c7af414" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34333 } }
I20260621 07:51:59.897506 15949 raft_consensus.cc:374] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:59.897584 15949 raft_consensus.cc:729] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58fb064fb0a46e3b9cf75822c7af414, State: Initialized, Role: FOLLOWER
I20260621 07:51:59.898056 15949 consensus_queue.cc:260] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58fb064fb0a46e3b9cf75822c7af414" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34333 } }
I20260621 07:51:59.898209 15949 raft_consensus.cc:388] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:59.898296 15949 raft_consensus.cc:482] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:59.898449 15949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:59.899396 15949 raft_consensus.cc:504] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58fb064fb0a46e3b9cf75822c7af414" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34333 } }
I20260621 07:51:59.899845 15949 leader_election.cc:302] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e58fb064fb0a46e3b9cf75822c7af414; no voters:
I20260621 07:51:59.900219 15949 leader_election.cc:288] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:59.900350 15954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:59.900709 15954 raft_consensus.cc:686] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [term 1 LEADER]: Becoming Leader. State: Replica: e58fb064fb0a46e3b9cf75822c7af414, State: Running, Role: LEADER
I20260621 07:51:59.901050 15954 consensus_queue.cc:237] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58fb064fb0a46e3b9cf75822c7af414" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34333 } }
I20260621 07:51:59.901634 15949 sys_catalog.cc:564] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:59.902369 15954 sys_catalog.cc:455] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e58fb064fb0a46e3b9cf75822c7af414. Latest consensus state: current_term: 1 leader_uuid: "e58fb064fb0a46e3b9cf75822c7af414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58fb064fb0a46e3b9cf75822c7af414" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34333 } } }
I20260621 07:51:59.902628 15954 sys_catalog.cc:458] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:59.902725 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:59.888164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44702 (local address 127.12.12.62:34333)
0621 07:51:59.888617 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:59.888634 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:59.888673 (+ 39us) server_negotiation.cc:407] Connection header received
0621 07:51:59.889340 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:59.889358 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:59.889766 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:59.890057 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:59.891075 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:59.892116 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:59.893290 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:59.893824 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:59.893964 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:59.897063 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:59.897069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:59.897084 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:59.897122 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:59.899199 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:59.899700 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:59.899706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:59.899713 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:59.899864 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:59.900181 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:59.900185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:59.900187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:59.900731 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:59.900961 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:59.901308 (+ 347us) server_negotiation.cc:299] Negotiation successful
0621 07:51:59.901620 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":153,"threads_started":1}
I20260621 07:51:59.902807 15955 sys_catalog.cc:455] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e58fb064fb0a46e3b9cf75822c7af414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58fb064fb0a46e3b9cf75822c7af414" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34333 } } }
I20260621 07:51:59.903105 15963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:59.903129 15955 sys_catalog.cc:458] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:59.906368 15963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:59.912204 15963 catalog_manager.cc:1347] Generated new cluster ID: 4622df1bbb10402ea13e9494ca62c4be
I20260621 07:51:59.912312 15963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:59.931618 15963 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:59.932628 15963 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:59.937556 15963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e58fb064fb0a46e3b9cf75822c7af414: Generated new TSK 0
I20260621 07:51:59.938345 15963 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:51:59.975628 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:34333
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:00.095247 15975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:00.095535 15975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:00.095588 15975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:00.099840 15975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:00.099915 15975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:00.100016 15975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:52:00.106014 15975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:34333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:00.107241 15975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:00.108332 15975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:00.118533 15980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:00.118535 15983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:00.118533 15981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:00.118882 15975 server_base.cc:1034] running on GCE node
I20260621 07:52:00.119318 15975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:00.120106 15975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:00.121300 15975 hybrid_clock.cc:648] HybridClock initialized: now 1782028320121261 us; error 50 us; skew 500 ppm
Jun 21 07:52:00 dist-test-slave-lbpt krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028320, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:00.124269 15975 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:52:00.125214 15975 webserver.cc:460] Webserver started at http://127.12.12.1:33109/ using document root <none> and password file <none>
I20260621 07:52:00.125805 15975 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:00.125870 15975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:00.126121 15975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:00.128360 15975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "5d1133e099f1415da7709982a14f56ff"
format_stamp: "Formatted at 2026-06-21 07:52:00 on dist-test-slave-lbpt"
server_key: "f68ce015947d52ae9d6001229e9ba7b3"
server_key_iv: "284af06223f4b727ebbf0af2c265418f"
server_key_version: "encryptionkey@0"
I20260621 07:52:00.128943 15975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "5d1133e099f1415da7709982a14f56ff"
format_stamp: "Formatted at 2026-06-21 07:52:00 on dist-test-slave-lbpt"
server_key: "f68ce015947d52ae9d6001229e9ba7b3"
server_key_iv: "284af06223f4b727ebbf0af2c265418f"
server_key_version: "encryptionkey@0"
I20260621 07:52:00.132900 15975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:52:00.135535 15990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:00.136806 15975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:52:00.136904 15975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "5d1133e099f1415da7709982a14f56ff"
format_stamp: "Formatted at 2026-06-21 07:52:00 on dist-test-slave-lbpt"
server_key: "f68ce015947d52ae9d6001229e9ba7b3"
server_key_iv: "284af06223f4b727ebbf0af2c265418f"
server_key_version: "encryptionkey@0"
I20260621 07:52:00.137045 15975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:00.150204 15975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:00.154072 15975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:00.154277 15975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:00.154954 15975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:00.155773 15975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:00.155823 15975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:00.155864 15975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:00.155879 15975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:00.164719 15975 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:34505
I20260621 07:52:00.164788 16103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:34505 every 8 connection(s)
I20260621 07:52:00.165670 15975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:52:00.172734 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 15975
I20260621 07:52:00.172835 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:52:00.173146 12336 external_mini_cluster.cc:1428] Setting key dca6ca3fbe577884b74a2b08b4b18d99
Jun 21 07:52:00 dist-test-slave-lbpt krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028320, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:00.179989 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:00.167620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:40001 (local address 127.12.12.62:34333)
0621 07:52:00.167786 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:00.167791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:00.169021 (+ 1230us) server_negotiation.cc:407] Connection header received
0621 07:52:00.169843 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:00.169845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:00.169891 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:00.169969 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:00.171257 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:00.171788 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:00.173034 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:00.173319 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:00.173438 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:00.175912 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:00.175916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:00.175921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:00.175952 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:00.177575 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:00.178231 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:00.178234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:00.178236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:00.178318 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:00.178746 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:00.178749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:00.178758 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:00.178957 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:00.179045 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:00.179742 (+ 697us) server_negotiation.cc:299] Negotiation successful
0621 07:52:00.179858 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 07:52:00.181354 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:00.168044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:34333 (local address 127.12.12.1:40001)
0621 07:52:00.168852 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:00.168905 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:00.169664 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:00.170152 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:00.170158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:00.170608 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:00.171099 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:00.171125 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:00.171906 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:00.171909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:00.172871 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:00.172886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:00.173478 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:00.173483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:00.173669 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:00.174256 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:00.174272 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:00.175745 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:00.177713 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:00.177721 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:00.177741 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:00.178115 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:00.178441 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:00.178444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:00.178448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:00.178644 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:00.179076 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:00.179087 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:00.179453 (+ 366us) client_negotiation.cc:769] Sending connection context
0621 07:52:00.179684 (+ 231us) client_negotiation.cc:240] Negotiation successful
0621 07:52:00.180090 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":107,"threads_started":1}
I20260621 07:52:00.182842 16104 heartbeater.cc:347] Connected to a master server at 127.12.12.62:34333
I20260621 07:52:00.183125 16104 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:00.183775 16104 heartbeater.cc:511] Master 127.12.12.62:34333 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
I20260621 07:52:00.185647 15913 ts_manager.cc:194] Registered new tserver with Master: 5d1133e099f1415da7709982a14f56ff (127.12.12.1:34505)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
I20260621 07:52:00.187201 15913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:40001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:52:00.201625 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:34333
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:00.314769 16109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:00.315042 16109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:00.315129 16109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:00.319454 16109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:00.319526 16109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:00.319628 16109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:52:00.325273 16109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:34333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:00.326557 16109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:00.327636 16109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:00.337715 16115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:00.337913 16109 server_base.cc:1034] running on GCE node
W20260621 07:52:00.337715 16117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:00.337738 16114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:00.338563 16109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:00.339216 16109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:00.340404 16109 hybrid_clock.cc:648] HybridClock initialized: now 1782028320340384 us; error 32 us; skew 500 ppm
Jun 21 07:52:00 dist-test-slave-lbpt krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028320, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:00.343423 16109 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:52:00.344417 16109 webserver.cc:460] Webserver started at http://127.12.12.2:41731/ using document root <none> and password file <none>
I20260621 07:52:00.344969 16109 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:00.345036 16109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:00.345203 16109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:00.347337 16109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "c16ac41e8060447a9db9a910d2aa63d2"
format_stamp: "Formatted at 2026-06-21 07:52:00 on dist-test-slave-lbpt"
server_key: "6408355f3528439e3fddd280cf811d79"
server_key_iv: "a8c02f4398d7741ed79f14ac8a1f52ef"
server_key_version: "encryptionkey@0"
I20260621 07:52:00.347848 16109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "c16ac41e8060447a9db9a910d2aa63d2"
format_stamp: "Formatted at 2026-06-21 07:52:00 on dist-test-slave-lbpt"
server_key: "6408355f3528439e3fddd280cf811d79"
server_key_iv: "a8c02f4398d7741ed79f14ac8a1f52ef"
server_key_version: "encryptionkey@0"
I20260621 07:52:00.351624 16109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:52:00.354565 16124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:00.355758 16109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 07:52:00.355885 16109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "c16ac41e8060447a9db9a910d2aa63d2"
format_stamp: "Formatted at 2026-06-21 07:52:00 on dist-test-slave-lbpt"
server_key: "6408355f3528439e3fddd280cf811d79"
server_key_iv: "a8c02f4398d7741ed79f14ac8a1f52ef"
server_key_version: "encryptionkey@0"
I20260621 07:52:00.356003 16109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:00.367184 16109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:00.370297 16109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:00.370580 16109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:00.371230 16109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:00.372087 16109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:00.372133 16109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:00.372174 16109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:00.372227 16109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:00.381641 16109 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:37943
I20260621 07:52:00.381661 16237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:37943 every 8 connection(s)
I20260621 07:52:00.382722 16109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:52:00.388890 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 16109
I20260621 07:52:00.389050 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:52:00.389364 12336 external_mini_cluster.cc:1428] Setting key 4e221f751f0269b415f7f8aae5ab3753
Jun 21 07:52:00 dist-test-slave-lbpt krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028320, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:00.397724 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:00.384736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:47631 (local address 127.12.12.62:34333)
0621 07:52:00.384893 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:00.384897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:00.385811 (+ 914us) server_negotiation.cc:407] Connection header received
0621 07:52:00.387301 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:00.387304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:00.387355 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:00.387459 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:00.388815 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:00.389629 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:00.390987 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:00.391212 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:00.391331 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:00.393987 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:00.393991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:00.393993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:00.394018 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:00.395584 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:00.396163 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:00.396167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:00.396170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:00.396249 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:00.396604 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:00.396608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:00.396610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:00.396802 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:00.396909 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:00.397420 (+ 511us) server_negotiation.cc:299] Negotiation successful
0621 07:52:00.397556 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 07:52:00.398697 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:00.385130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:34333 (local address 127.12.12.2:47631)
0621 07:52:00.385630 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:00.385677 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:00.387071 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:00.387680 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:00.387688 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:00.388079 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:00.388640 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:00.388655 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:00.389771 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:00.389773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:00.390841 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:00.390852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:00.391393 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:00.391398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:00.391562 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:00.392250 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:00.392267 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:00.393794 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:00.395714 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:00.395721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:00.395745 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:00.396044 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:00.396342 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:00.396346 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:00.396348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:00.396498 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:00.396908 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:00.396917 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:00.397176 (+ 259us) client_negotiation.cc:769] Sending connection context
0621 07:52:00.397381 (+ 205us) client_negotiation.cc:240] Negotiation successful
0621 07:52:00.397733 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
I20260621 07:52:00.400228 16238 heartbeater.cc:347] Connected to a master server at 127.12.12.62:34333
I20260621 07:52:00.400777 16238 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:00.401391 16238 heartbeater.cc:511] Master 127.12.12.62:34333 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
I20260621 07:52:00.402777 15913 ts_manager.cc:194] Registered new tserver with Master: c16ac41e8060447a9db9a910d2aa63d2 (127.12.12.2:37943)
I20260621 07:52:00.403508 15913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:47631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:52:00.423239 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:34333
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:00.538955 16243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:00.539316 16243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:00.539386 16243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:00.544211 16243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:00.544276 16243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:00.544365 16243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:52:00.550549 16243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:34333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:00.551633 16243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:00.552738 16243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:00.562587 16249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:00.562583 16251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:00.562839 16248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:00.562893 16243 server_base.cc:1034] running on GCE node
I20260621 07:52:00.563398 16243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:00.564050 16243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:00.565232 16243 hybrid_clock.cc:648] HybridClock initialized: now 1782028320565203 us; error 51 us; skew 500 ppm
Jun 21 07:52:00 dist-test-slave-lbpt krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028320, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:00.568053 16243 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:52:00.568955 16243 webserver.cc:460] Webserver started at http://127.12.12.3:41249/ using document root <none> and password file <none>
I20260621 07:52:00.569540 16243 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:00.569628 16243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:00.569798 16243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:00.571818 16243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "b2e67cf757bf44fc92cc2dcb9bc41346"
format_stamp: "Formatted at 2026-06-21 07:52:00 on dist-test-slave-lbpt"
server_key: "4d160b478dbde222e3b582f9d74d8c9d"
server_key_iv: "d29c4c6d3db771b241a2fdc498894508"
server_key_version: "encryptionkey@0"
I20260621 07:52:00.572312 16243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "b2e67cf757bf44fc92cc2dcb9bc41346"
format_stamp: "Formatted at 2026-06-21 07:52:00 on dist-test-slave-lbpt"
server_key: "4d160b478dbde222e3b582f9d74d8c9d"
server_key_iv: "d29c4c6d3db771b241a2fdc498894508"
server_key_version: "encryptionkey@0"
I20260621 07:52:00.575965 16243 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:52:00.578394 16258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:00.579785 16243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:00.579877 16243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "b2e67cf757bf44fc92cc2dcb9bc41346"
format_stamp: "Formatted at 2026-06-21 07:52:00 on dist-test-slave-lbpt"
server_key: "4d160b478dbde222e3b582f9d74d8c9d"
server_key_iv: "d29c4c6d3db771b241a2fdc498894508"
server_key_version: "encryptionkey@0"
I20260621 07:52:00.579964 16243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:00.596336 16243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:00.599903 16243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:00.600131 16243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:00.600693 16243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:00.601548 16243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:00.601594 16243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:00.601635 16243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:00.601657 16243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:00.611271 16243 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:45147
I20260621 07:52:00.611352 16371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:45147 every 8 connection(s)
I20260621 07:52:00.612215 16243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:52:00.620579 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 16243
I20260621 07:52:00.620752 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:52:00.621114 12336 external_mini_cluster.cc:1428] Setting key 673c216da797c808c99fa8d3fd67a6b7
Jun 21 07:52:00 dist-test-slave-lbpt krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028320, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:00.626307 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:00.614178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:53675 (local address 127.12.12.62:34333)
0621 07:52:00.614341 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:00.614345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:00.615214 (+ 869us) server_negotiation.cc:407] Connection header received
0621 07:52:00.616637 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:00.616640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:00.616702 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:00.616810 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:00.618004 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:00.618623 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:00.619798 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:00.620044 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:00.620160 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:00.622920 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:00.622924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:00.622926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:00.622959 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:00.624460 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:00.624925 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:00.624928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:00.624929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:00.625008 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:00.625313 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:00.625316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:00.625317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:00.625495 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:00.625616 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:00.626090 (+ 474us) server_negotiation.cc:299] Negotiation successful
0621 07:52:00.626187 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 07:52:00.627202 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:00.614577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:34333 (local address 127.12.12.3:53675)
0621 07:52:00.615049 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:00.615094 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:00.616413 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:00.616997 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:00.617003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:00.617369 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:00.617843 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:00.617856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:00.618775 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:00.618778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:00.619666 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:00.619675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:00.620201 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:00.620205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:00.620379 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:00.621061 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:00.621077 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:00.622755 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:00.624574 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:00.624579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:00.624592 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:00.624839 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:00.625084 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:00.625087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:00.625088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:00.625224 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:00.625623 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:00.625629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:00.625860 (+ 231us) client_negotiation.cc:769] Sending connection context
0621 07:52:00.626044 (+ 184us) client_negotiation.cc:240] Negotiation successful
0621 07:52:00.626307 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
I20260621 07:52:00.628392 16372 heartbeater.cc:347] Connected to a master server at 127.12.12.62:34333
I20260621 07:52:00.628660 16372 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:00.629171 16372 heartbeater.cc:511] Master 127.12.12.62:34333 requested a full tablet report, sending...
I20260621 07:52:00.630303 15913 ts_manager.cc:194] Registered new tserver with Master: b2e67cf757bf44fc92cc2dcb9bc41346 (127.12.12.3:45147)
I20260621 07:52:00.630935 15913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:53675
I20260621 07:52:00.635587 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 07:52:00 dist-test-slave-lbpt krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 07:52:00 dist-test-slave-lbpt krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.1@KRBTEST.COM
I20260621 07:52:00.662333 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:00.652367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52640 (local address 127.12.12.1:34505)
0621 07:52:00.652671 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:00.652697 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:00.652722 (+ 25us) server_negotiation.cc:407] Connection header received
0621 07:52:00.652782 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:00.652786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:00.653054 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:00.653227 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:00.654089 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:00.655041 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:00.655861 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:00.656236 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:00.656318 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:00.658814 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:00.658823 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:00.658837 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:00.658863 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:00.660603 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:00.661013 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:00.661019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:00.661020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:00.661102 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:00.661327 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:00.661329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:00.661330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:00.661605 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:00.661765 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:00.661952 (+ 187us) server_negotiation.cc:299] Negotiation successful
0621 07:52:00.662129 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
Jun 21 07:52:00 dist-test-slave-lbpt krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:00.674240 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:00.666162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.12.12.62:34333)
0621 07:52:00.666299 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:00.666303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:00.666315 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:52:00.666352 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:00.666354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:00.666392 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:00.666516 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:00.667169 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:00.667690 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:00.668451 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:00.668664 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:00.668737 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:00.671132 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:00.671135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:00.671137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:00.671161 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:00.672585 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:00.672987 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:00.672990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:00.672991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:00.673064 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:00.673327 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:00.673329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:00.673330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:00.673560 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:00.673661 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:00.673924 (+ 263us) server_negotiation.cc:299] Negotiation successful
0621 07:52:00.674019 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260621 07:52:00.674650 15913 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44712
I20260621 07:52:00.675279 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 15975
I20260621 07:52:00.681618 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 16109
I20260621 07:52:00.687451 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 16243
I20260621 07:52:00.692879 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 15881
2026-06-21T07:52:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2187 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:52:00 dist-test-slave-lbpt krb5kdc[16384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:52:00 dist-test-slave-lbpt krb5kdc[16384](info): set up 2 sockets
Jun 21 07:52:00 dist-test-slave-lbpt krb5kdc[16384](info): commencing operation
krb5kdc: starting...
W20260621 07:52:01.734681 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:52:01 dist-test-slave-lbpt krb5kdc[16384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:52:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:52:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:52:01.864107 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:37297
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:37297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:01.979422 16397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:01.979794 16397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:01.979877 16397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:01.984135 16397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:01.984253 16397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:01.984292 16397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:01.984326 16397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:01.984364 16397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:52:01.990092 16397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:37297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:37297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:01.991348 16397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:01.992726 16397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:02.004693 16403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:02.004706 16402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:02.004698 16405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:02.004937 16397 server_base.cc:1034] running on GCE node
I20260621 07:52:02.005641 16397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:02.007159 16397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:02.008365 16397 hybrid_clock.cc:648] HybridClock initialized: now 1782028322008318 us; error 64 us; skew 500 ppm
Jun 21 07:52:02 dist-test-slave-lbpt krb5kdc[16384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028322, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:02.014307 16397 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:52:02.015192 16397 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:02.015264 16397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:02.015440 16397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:02.017542 16397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "cf1c81564004494fbba7b0976906d5eb"
format_stamp: "Formatted at 2026-06-21 07:52:02 on dist-test-slave-lbpt"
server_key: "2075bea333f1546b0f62465d87a4f446"
server_key_iv: "7bdf83219b087f64cf2778e278114470"
server_key_version: "encryptionkey@0"
I20260621 07:52:02.018078 16397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "cf1c81564004494fbba7b0976906d5eb"
format_stamp: "Formatted at 2026-06-21 07:52:02 on dist-test-slave-lbpt"
server_key: "2075bea333f1546b0f62465d87a4f446"
server_key_iv: "7bdf83219b087f64cf2778e278114470"
server_key_version: "encryptionkey@0"
I20260621 07:52:02.022091 16397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 07:52:02.024823 16411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:02.026158 16397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:52:02.026316 16397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "cf1c81564004494fbba7b0976906d5eb"
format_stamp: "Formatted at 2026-06-21 07:52:02 on dist-test-slave-lbpt"
server_key: "2075bea333f1546b0f62465d87a4f446"
server_key_iv: "7bdf83219b087f64cf2778e278114470"
server_key_version: "encryptionkey@0"
I20260621 07:52:02.026486 16397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:02.040768 16397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:02.044298 16397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:02.044529 16397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:02.051846 16397 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:37297
I20260621 07:52:02.052111 16463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:37297 every 8 connection(s)
I20260621 07:52:02.052786 16397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:52:02.055791 16464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:02.061439 16464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb: Bootstrap starting.
I20260621 07:52:02.061499 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 16397
I20260621 07:52:02.061614 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:52:02.061896 12336 external_mini_cluster.cc:1428] Setting key 0a5f948919db7e4125486c77ad8ede6c
I20260621 07:52:02.064272 16464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:02.065052 16464 log.cc:824] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:02.067238 16464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb: No bootstrap required, opened a new log
Jun 21 07:52:02 dist-test-slave-lbpt krb5kdc[16384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:02.071849 16464 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf1c81564004494fbba7b0976906d5eb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 37297 } }
I20260621 07:52:02.072122 16464 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:02.072198 16464 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1c81564004494fbba7b0976906d5eb, State: Initialized, Role: FOLLOWER
I20260621 07:52:02.072749 16464 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf1c81564004494fbba7b0976906d5eb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 37297 } }
I20260621 07:52:02.072918 16464 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:52:02.072991 16464 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:52:02.073096 16464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:02.074360 16464 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf1c81564004494fbba7b0976906d5eb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 37297 } }
I20260621 07:52:02.074829 16464 leader_election.cc:302] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf1c81564004494fbba7b0976906d5eb; no voters:
I20260621 07:52:02.075213 16464 leader_election.cc:288] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:52:02.075335 16469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:02.075562 16469 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [term 1 LEADER]: Becoming Leader. State: Replica: cf1c81564004494fbba7b0976906d5eb, State: Running, Role: LEADER
I20260621 07:52:02.075923 16469 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf1c81564004494fbba7b0976906d5eb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 37297 } }
I20260621 07:52:02.076539 16464 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:52:02.077270 16471 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf1c81564004494fbba7b0976906d5eb. Latest consensus state: current_term: 1 leader_uuid: "cf1c81564004494fbba7b0976906d5eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf1c81564004494fbba7b0976906d5eb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 37297 } } }
I20260621 07:52:02.077426 16471 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:02.077692 16470 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf1c81564004494fbba7b0976906d5eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf1c81564004494fbba7b0976906d5eb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 37297 } } }
I20260621 07:52:02.078023 16478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:52:02.077778 16470 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:02.078781 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:02.063351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58444 (local address 127.12.12.62:37297)
0621 07:52:02.063774 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:02.063791 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:02.063827 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:52:02.064558 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:02.064578 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:02.064933 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:02.065294 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:02.066128 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.067203 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.068377 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.068850 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.068970 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:02.071960 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:02.071965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:02.071981 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:02.072010 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:02.074214 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:02.074788 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:02.074795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:02.074801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:02.074964 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:02.075440 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:02.075444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:02.075447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:02.076013 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:02.076213 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:02.076836 (+ 623us) server_negotiation.cc:299] Negotiation successful
0621 07:52:02.077163 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":144,"threads_started":1}
I20260621 07:52:02.081318 16478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:52:02.087565 16478 catalog_manager.cc:1347] Generated new cluster ID: 9eb96f5ef0034c2b8251dea0b93649a7
I20260621 07:52:02.087726 16478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:52:02.116829 16478 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:52:02.118059 16478 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:52:02.132318 16478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb: Generated new TSK 0
I20260621 07:52:02.133119 16478 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:52:02.170181 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:37297
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:02.292719 16490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:02.293015 16490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:02.293062 16490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:02.297432 16490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:02.297511 16490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:02.297610 16490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:52:02.303268 16490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:37297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:02.304407 16490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:02.305467 16490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:02.315173 16496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:02.315188 16495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:02.315177 16498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:02.315548 16490 server_base.cc:1034] running on GCE node
I20260621 07:52:02.316054 16490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:02.316682 16490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:02.317845 16490 hybrid_clock.cc:648] HybridClock initialized: now 1782028322317828 us; error 36 us; skew 500 ppm
Jun 21 07:52:02 dist-test-slave-lbpt krb5kdc[16384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028322, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:02.320822 16490 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:52:02.321532 16490 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:02.321597 16490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:02.321762 16490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:02.323844 16490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "fc26fdec3ed34f8c8d9826274ea2b804"
format_stamp: "Formatted at 2026-06-21 07:52:02 on dist-test-slave-lbpt"
server_key: "2d944301c1f4391a1b339b5b92c76aa3"
server_key_iv: "2b958266da10c433e025dace73def545"
server_key_version: "encryptionkey@0"
I20260621 07:52:02.324360 16490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "fc26fdec3ed34f8c8d9826274ea2b804"
format_stamp: "Formatted at 2026-06-21 07:52:02 on dist-test-slave-lbpt"
server_key: "2d944301c1f4391a1b339b5b92c76aa3"
server_key_iv: "2b958266da10c433e025dace73def545"
server_key_version: "encryptionkey@0"
I20260621 07:52:02.328027 16490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:52:02.330601 16504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:02.331954 16490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:52:02.332095 16490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "fc26fdec3ed34f8c8d9826274ea2b804"
format_stamp: "Formatted at 2026-06-21 07:52:02 on dist-test-slave-lbpt"
server_key: "2d944301c1f4391a1b339b5b92c76aa3"
server_key_iv: "2b958266da10c433e025dace73def545"
server_key_version: "encryptionkey@0"
I20260621 07:52:02.332201 16490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:02.347860 16490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:02.351352 16490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:02.351541 16490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:02.352174 16490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:02.353019 16490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:02.353072 16490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:02.353137 16490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:02.353169 16490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:02.362550 16490 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:39747
I20260621 07:52:02.362627 16617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:39747 every 8 connection(s)
I20260621 07:52:02.363224 16490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:52:02.367152 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 16490
I20260621 07:52:02.367308 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:52:02.367581 12336 external_mini_cluster.cc:1428] Setting key 07be692bebde13303119b171b8ed4089
Jun 21 07:52:02 dist-test-slave-lbpt krb5kdc[16384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028322, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
I20260621 07:52:02.379298 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:02.365006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:42405 (local address 127.12.12.62:37297)
0621 07:52:02.365143 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:02.365147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:02.366046 (+ 899us) server_negotiation.cc:407] Connection header received
0621 07:52:02.367577 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:02.367580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:02.367642 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:02.367733 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:02.369075 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.369613 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.371257 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.371678 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.371811 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:02.374777 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:02.374784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:02.374791 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:02.374829 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:02.376651 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:02.377450 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:02.377453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:02.377454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:02.377539 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:02.378055 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:02.378058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:02.378060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:02.378259 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:02.378339 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:02.378944 (+ 605us) server_negotiation.cc:299] Negotiation successful
0621 07:52:02.379148 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:52:02.380126 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:02.365416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:37297 (local address 127.12.12.1:42405)
0621 07:52:02.365887 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:02.365931 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:02.367238 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:02.367959 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:02.367970 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:02.368368 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:02.368896 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:02.368911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.369750 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.369757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:02.371122 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:02.371133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.371838 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.371841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:02.371985 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:02.372632 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:02.372649 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:02.374584 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:02.376828 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:02.376838 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:02.376864 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:02.377275 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:02.377668 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:02.377671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:02.377673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:02.377959 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:02.378368 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:02.378375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:02.378704 (+ 329us) client_negotiation.cc:769] Sending connection context
0621 07:52:02.378870 (+ 166us) client_negotiation.cc:240] Negotiation successful
0621 07:52:02.379210 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
I20260621 07:52:02.381598 16618 heartbeater.cc:347] Connected to a master server at 127.12.12.62:37297
I20260621 07:52:02.381917 16618 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:02.382488 16618 heartbeater.cc:511] Master 127.12.12.62:37297 requested a full tablet report, sending...
I20260621 07:52:02.384300 16428 ts_manager.cc:194] Registered new tserver with Master: fc26fdec3ed34f8c8d9826274ea2b804 (127.12.12.1:39747)
I20260621 07:52:02.385536 16428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:42405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:52:02.402280 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:37297
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:02.516693 16623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:02.516986 16623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:02.517035 16623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:02.521303 16623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:02.521373 16623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:02.521487 16623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:52:02.527112 16623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:37297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:02.528368 16623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:02.529453 16623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:02.539379 16628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:02.539384 16631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:02.539384 16629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:02.539791 16623 server_base.cc:1034] running on GCE node
I20260621 07:52:02.540277 16623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:02.540978 16623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:02.542203 16623 hybrid_clock.cc:648] HybridClock initialized: now 1782028322542178 us; error 37 us; skew 500 ppm
Jun 21 07:52:02 dist-test-slave-lbpt krb5kdc[16384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028322, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:02.546223 16623 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:52:02.547314 16623 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:02.547418 16623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:02.547708 16623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:02.549965 16623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "4b977f65b3b44bfcb5bbc7a1edb102fd"
format_stamp: "Formatted at 2026-06-21 07:52:02 on dist-test-slave-lbpt"
server_key: "f9eb91e8fe6f4beebf77a1c98b8bfd77"
server_key_iv: "59f8da7c2e738afbe6351c80c394990d"
server_key_version: "encryptionkey@0"
I20260621 07:52:02.550608 16623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "4b977f65b3b44bfcb5bbc7a1edb102fd"
format_stamp: "Formatted at 2026-06-21 07:52:02 on dist-test-slave-lbpt"
server_key: "f9eb91e8fe6f4beebf77a1c98b8bfd77"
server_key_iv: "59f8da7c2e738afbe6351c80c394990d"
server_key_version: "encryptionkey@0"
I20260621 07:52:02.554378 16623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260621 07:52:02.556993 16637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:02.558279 16623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:52:02.558389 16623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "4b977f65b3b44bfcb5bbc7a1edb102fd"
format_stamp: "Formatted at 2026-06-21 07:52:02 on dist-test-slave-lbpt"
server_key: "f9eb91e8fe6f4beebf77a1c98b8bfd77"
server_key_iv: "59f8da7c2e738afbe6351c80c394990d"
server_key_version: "encryptionkey@0"
I20260621 07:52:02.558614 16623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:02.572732 16623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:02.576360 16623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:02.576529 16623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:02.577342 16623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:02.578496 16623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:02.578557 16623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:02.578599 16623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:02.578614 16623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:02.588483 16623 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:36113
I20260621 07:52:02.588517 16750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:36113 every 8 connection(s)
I20260621 07:52:02.589303 16623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:52:02.599681 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 16623
I20260621 07:52:02.599805 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:52:02.600122 12336 external_mini_cluster.cc:1428] Setting key d3c1bbc2d44561c4955d8be3a1a1d75d
Jun 21 07:52:02 dist-test-slave-lbpt krb5kdc[16384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028322, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:02.604923 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:02.591415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:51121 (local address 127.12.12.62:37297)
0621 07:52:02.591601 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:02.591606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:02.592625 (+ 1019us) server_negotiation.cc:407] Connection header received
0621 07:52:02.594011 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:02.594015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:02.594067 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:02.594148 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:02.595620 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.596207 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.597520 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.597785 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.597887 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:02.600985 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:02.600990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:02.600993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:02.601031 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:02.602702 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:02.603281 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:02.603284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:02.603286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:02.603380 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:02.603759 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:02.603762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:02.603764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:02.603957 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:02.604064 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:02.604625 (+ 561us) server_negotiation.cc:299] Negotiation successful
0621 07:52:02.604757 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 07:52:02.605877 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:02.591878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:37297 (local address 127.12.12.2:51121)
0621 07:52:02.592446 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:02.592496 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:02.593773 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:02.594338 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:02.594346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:02.594903 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:02.595432 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:02.595448 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.596349 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.596352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:02.597362 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:02.597375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.597952 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.597955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:02.598126 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:02.598820 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:02.598838 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:02.600794 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:02.602841 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:02.602847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:02.602866 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:02.603166 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:02.603477 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:02.603481 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:02.603486 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:02.603651 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:02.604064 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:02.604073 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:02.604362 (+ 289us) client_negotiation.cc:769] Sending connection context
0621 07:52:02.604571 (+ 209us) client_negotiation.cc:240] Negotiation successful
0621 07:52:02.604928 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":145,"threads_started":1}
I20260621 07:52:02.607185 16751 heartbeater.cc:347] Connected to a master server at 127.12.12.62:37297
I20260621 07:52:02.607458 16751 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:02.607963 16751 heartbeater.cc:511] Master 127.12.12.62:37297 requested a full tablet report, sending...
I20260621 07:52:02.609135 16428 ts_manager.cc:194] Registered new tserver with Master: 4b977f65b3b44bfcb5bbc7a1edb102fd (127.12.12.2:36113)
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
I20260621 07:52:02.610014 16428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:51121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:52:02.631178 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:37297
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:02.746479 16756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:02.746767 16756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:02.746815 16756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:02.750847 16756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:02.750909 16756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:02.750972 16756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:52:02.756281 16756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:37297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:02.757368 16756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:02.758364 16756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:02.768213 16762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:02.768208 16761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:02.768203 16764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:02.768479 16756 server_base.cc:1034] running on GCE node
I20260621 07:52:02.768970 16756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:02.769584 16756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:02.770740 16756 hybrid_clock.cc:648] HybridClock initialized: now 1782028322770711 us; error 46 us; skew 500 ppm
Jun 21 07:52:02 dist-test-slave-lbpt krb5kdc[16384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028322, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:02.773530 16756 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:52:02.774281 16756 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:02.774344 16756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:02.774580 16756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:02.776659 16756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "87598af7bb284c969bdc9f8b94bb0044"
format_stamp: "Formatted at 2026-06-21 07:52:02 on dist-test-slave-lbpt"
server_key: "f753d37cd4594c0a3048b3c844a22f41"
server_key_iv: "7b1b650487924dc5ffc85ae97ef8b1fb"
server_key_version: "encryptionkey@0"
I20260621 07:52:02.777200 16756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "87598af7bb284c969bdc9f8b94bb0044"
format_stamp: "Formatted at 2026-06-21 07:52:02 on dist-test-slave-lbpt"
server_key: "f753d37cd4594c0a3048b3c844a22f41"
server_key_iv: "7b1b650487924dc5ffc85ae97ef8b1fb"
server_key_version: "encryptionkey@0"
I20260621 07:52:02.781256 16756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 07:52:02.783943 16770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:02.785313 16756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:52:02.785421 16756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "87598af7bb284c969bdc9f8b94bb0044"
format_stamp: "Formatted at 2026-06-21 07:52:02 on dist-test-slave-lbpt"
server_key: "f753d37cd4594c0a3048b3c844a22f41"
server_key_iv: "7b1b650487924dc5ffc85ae97ef8b1fb"
server_key_version: "encryptionkey@0"
I20260621 07:52:02.785609 16756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:02.800973 16756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:02.804113 16756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:02.804316 16756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:02.804909 16756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:02.805766 16756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:02.805814 16756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:02.805852 16756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:02.805867 16756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:02.815137 16756 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:33777
I20260621 07:52:02.815155 16883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:33777 every 8 connection(s)
I20260621 07:52:02.815773 16756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:52:02.818377 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 16756
I20260621 07:52:02.818532 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:52:02.818832 12336 external_mini_cluster.cc:1428] Setting key dd79f956fe7366201a6299e26e88056b
Jun 21 07:52:02 dist-test-slave-lbpt krb5kdc[16384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028322, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:02.829480 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:02.817541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:51989 (local address 127.12.12.62:37297)
0621 07:52:02.817681 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:02.817685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:02.818568 (+ 883us) server_negotiation.cc:407] Connection header received
0621 07:52:02.820214 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:02.820217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:02.820270 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:02.820356 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:02.821752 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.822274 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.823454 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.823661 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.823739 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:02.826105 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:02.826107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:02.826109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:02.826133 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:02.827657 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:02.828159 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:02.828162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:02.828163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:02.828238 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:02.828540 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:02.828542 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:02.828544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:02.828707 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:02.828780 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:02.829260 (+ 480us) server_negotiation.cc:299] Negotiation successful
0621 07:52:02.829357 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:52:02.830565 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:02.817884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:37297 (local address 127.12.12.3:51989)
0621 07:52:02.818352 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:02.818462 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:02.819863 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:02.820564 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:02.820574 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:02.821099 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:02.821627 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:02.821641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.822386 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.822390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:02.823346 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:02.823355 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.823790 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.823793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:02.823959 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:02.824500 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:02.824515 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:02.825948 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:02.827763 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:02.827768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:02.827781 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:02.828069 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:02.828315 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:02.828317 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:02.828319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:02.828455 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:02.828791 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:02.828796 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:02.829055 (+ 259us) client_negotiation.cc:769] Sending connection context
0621 07:52:02.829217 (+ 162us) client_negotiation.cc:240] Negotiation successful
0621 07:52:02.829474 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
I20260621 07:52:02.831844 16884 heartbeater.cc:347] Connected to a master server at 127.12.12.62:37297
I20260621 07:52:02.832137 16884 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:02.832652 16884 heartbeater.cc:511] Master 127.12.12.62:37297 requested a full tablet report, sending...
I20260621 07:52:02.833702 16428 ts_manager.cc:194] Registered new tserver with Master: 87598af7bb284c969bdc9f8b94bb0044 (127.12.12.3:33777)
I20260621 07:52:02.834266 16428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:51989
I20260621 07:52:02.845212 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:52:02.854733 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:02.847193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.12.12.62:37297)
0621 07:52:02.847338 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:02.847343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:02.847535 (+ 192us) server_negotiation.cc:407] Connection header received
0621 07:52:02.847685 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:02.847693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:02.847737 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:02.847824 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:02.848564 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.849099 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.849962 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.850190 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.850304 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:02.851737 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:02.851741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:02.851744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:02.851775 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:02.852909 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:02.853387 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:02.853392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:02.853394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:02.853480 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:02.853773 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:02.853776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:02.853779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:02.854013 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:02.854155 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:02.854404 (+ 249us) server_negotiation.cc:299] Negotiation successful
0621 07:52:02.854562 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 07:52:02.857523 16428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:52:02.860662 16428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:52:02.875905 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:02.870603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.12.12.1:39747)
0621 07:52:02.871026 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:02.871036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:02.871062 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:52:02.871143 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:02.871148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:02.871395 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:02.871580 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:02.871899 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.872765 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.874358 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.875144 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.875259 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:02.875441 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:02.875621 (+ 180us) server_negotiation.cc:299] Negotiation successful
0621 07:52:02.875734 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":127,"threads_started":1}
I20260621 07:52:02.875897 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:02.870076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:39747 (local address 127.0.0.1:38238)
0621 07:52:02.870805 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:02.870818 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:02.870927 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:02.871573 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:02.871579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:02.871600 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:02.871751 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:02.871760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.872935 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.872938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:02.874206 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:02.874216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.875286 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.875289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:02.875376 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:02.875420 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 07:52:02.875580 (+ 160us) client_negotiation.cc:240] Negotiation successful
0621 07:52:02.875722 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":114,"spinlock_wait_cycles":5120,"thread_start_us":95,"threads_started":1}
I20260621 07:52:02.876705 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:02.869777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.12.12.2:36113)
0621 07:52:02.870292 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:02.870305 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:02.870801 (+ 496us) server_negotiation.cc:407] Connection header received
0621 07:52:02.871116 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:02.871122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:02.871388 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:02.871529 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:02.871971 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.873192 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.874887 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.875759 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.875864 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:02.876012 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:02.876444 (+ 432us) server_negotiation.cc:299] Negotiation successful
0621 07:52:02.876557 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":159,"threads_started":1}
I20260621 07:52:02.877240 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:02.869618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:36113 (local address 127.0.0.1:57804)
0621 07:52:02.870649 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:02.870707 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:02.870927 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:02.871747 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:02.871751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:02.871769 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:02.871872 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:02.871878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.873344 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.873348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:02.874749 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:02.874760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.876080 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.876087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:02.876217 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:02.876252 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 07:52:02.877001 (+ 749us) client_negotiation.cc:240] Negotiation successful
0621 07:52:02.877092 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":117,"spinlock_wait_cycles":4352,"thread_start_us":116,"threads_started":1}
I20260621 07:52:02.878113 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:02.870956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:33777 (local address 127.0.0.1:46944)
0621 07:52:02.871231 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:02.871244 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:02.871340 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:02.871802 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:02.871808 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:02.871831 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:02.871914 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:02.871924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.873344 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.873348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:02.874749 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:02.874760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.877702 (+ 2942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.877706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:02.877793 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:02.877815 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:52:02.877893 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 07:52:02.877968 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":85,"thread_start_us":93,"threads_started":1}
I20260621 07:52:02.878450 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:02.870630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.12.12.3:33777)
0621 07:52:02.871221 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:02.871232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:02.871336 (+ 104us) server_negotiation.cc:407] Connection header received
0621 07:52:02.871468 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:02.871480 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:02.871649 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:02.871789 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:02.872027 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.873179 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.875526 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:02.877364 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:02.877492 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:02.877638 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:02.877950 (+ 312us) server_negotiation.cc:299] Negotiation successful
0621 07:52:02.878105 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":100,"threads_started":1}
I20260621 07:52:02.879590 16552 tablet_service.cc:1467] Processing CreateTablet for tablet d451d2f95842424a8f7ef18bcfb4cf54 (DEFAULT_TABLE table=test-table [id=889e275c127047548f43ccf2be71483b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:02.880264 16685 tablet_service.cc:1467] Processing CreateTablet for tablet d451d2f95842424a8f7ef18bcfb4cf54 (DEFAULT_TABLE table=test-table [id=889e275c127047548f43ccf2be71483b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:02.880805 16552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d451d2f95842424a8f7ef18bcfb4cf54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:02.881220 16685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d451d2f95842424a8f7ef18bcfb4cf54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:02.881165 16818 tablet_service.cc:1467] Processing CreateTablet for tablet d451d2f95842424a8f7ef18bcfb4cf54 (DEFAULT_TABLE table=test-table [id=889e275c127047548f43ccf2be71483b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:02.882107 16818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d451d2f95842424a8f7ef18bcfb4cf54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:02.888262 16901 tablet_bootstrap.cc:492] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd: Bootstrap starting.
I20260621 07:52:02.888737 16902 tablet_bootstrap.cc:492] T d451d2f95842424a8f7ef18bcfb4cf54 P fc26fdec3ed34f8c8d9826274ea2b804: Bootstrap starting.
I20260621 07:52:02.889091 16903 tablet_bootstrap.cc:492] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044: Bootstrap starting.
I20260621 07:52:02.891045 16902 tablet_bootstrap.cc:654] T d451d2f95842424a8f7ef18bcfb4cf54 P fc26fdec3ed34f8c8d9826274ea2b804: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:02.891045 16901 tablet_bootstrap.cc:654] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:02.891144 16903 tablet_bootstrap.cc:654] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:02.891994 16902 log.cc:824] T d451d2f95842424a8f7ef18bcfb4cf54 P fc26fdec3ed34f8c8d9826274ea2b804: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:02.892153 16903 log.cc:824] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:02.892231 16901 log.cc:824] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:02.894138 16903 tablet_bootstrap.cc:492] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044: No bootstrap required, opened a new log
I20260621 07:52:02.894136 16902 tablet_bootstrap.cc:492] T d451d2f95842424a8f7ef18bcfb4cf54 P fc26fdec3ed34f8c8d9826274ea2b804: No bootstrap required, opened a new log
I20260621 07:52:02.894136 16901 tablet_bootstrap.cc:492] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd: No bootstrap required, opened a new log
I20260621 07:52:02.894347 16903 ts_tablet_manager.cc:1397] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:52:02.894348 16902 ts_tablet_manager.cc:1397] T d451d2f95842424a8f7ef18bcfb4cf54 P fc26fdec3ed34f8c8d9826274ea2b804: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:52:02.894348 16901 ts_tablet_manager.cc:1397] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260621 07:52:02.897284 16903 raft_consensus.cc:348] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b977f65b3b44bfcb5bbc7a1edb102fd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 36113 } } peers { permanent_uuid: "87598af7bb284c969bdc9f8b94bb0044" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33777 } } peers { permanent_uuid: "fc26fdec3ed34f8c8d9826274ea2b804" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39747 } }
I20260621 07:52:02.897553 16903 raft_consensus.cc:374] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:02.897612 16903 raft_consensus.cc:729] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87598af7bb284c969bdc9f8b94bb0044, State: Initialized, Role: FOLLOWER
I20260621 07:52:02.898080 16903 consensus_queue.cc:260] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b977f65b3b44bfcb5bbc7a1edb102fd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 36113 } } peers { permanent_uuid: "87598af7bb284c969bdc9f8b94bb0044" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33777 } } peers { permanent_uuid: "fc26fdec3ed34f8c8d9826274ea2b804" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39747 } }
I20260621 07:52:02.898360 16901 raft_consensus.cc:348] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b977f65b3b44bfcb5bbc7a1edb102fd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 36113 } } peers { permanent_uuid: "87598af7bb284c969bdc9f8b94bb0044" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33777 } } peers { permanent_uuid: "fc26fdec3ed34f8c8d9826274ea2b804" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39747 } }
I20260621 07:52:02.898663 16901 raft_consensus.cc:374] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:02.898720 16901 raft_consensus.cc:729] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b977f65b3b44bfcb5bbc7a1edb102fd, State: Initialized, Role: FOLLOWER
I20260621 07:52:02.898826 16884 heartbeater.cc:503] Master 127.12.12.62:37297 was elected leader, sending a full tablet report...
I20260621 07:52:02.898743 16902 raft_consensus.cc:348] T d451d2f95842424a8f7ef18bcfb4cf54 P fc26fdec3ed34f8c8d9826274ea2b804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b977f65b3b44bfcb5bbc7a1edb102fd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 36113 } } peers { permanent_uuid: "87598af7bb284c969bdc9f8b94bb0044" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33777 } } peers { permanent_uuid: "fc26fdec3ed34f8c8d9826274ea2b804" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39747 } }
I20260621 07:52:02.898969 16902 raft_consensus.cc:374] T d451d2f95842424a8f7ef18bcfb4cf54 P fc26fdec3ed34f8c8d9826274ea2b804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:02.899026 16902 raft_consensus.cc:729] T d451d2f95842424a8f7ef18bcfb4cf54 P fc26fdec3ed34f8c8d9826274ea2b804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc26fdec3ed34f8c8d9826274ea2b804, State: Initialized, Role: FOLLOWER
I20260621 07:52:02.899225 16903 ts_tablet_manager.cc:1428] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:52:02.899155 16901 consensus_queue.cc:260] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b977f65b3b44bfcb5bbc7a1edb102fd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 36113 } } peers { permanent_uuid: "87598af7bb284c969bdc9f8b94bb0044" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33777 } } peers { permanent_uuid: "fc26fdec3ed34f8c8d9826274ea2b804" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39747 } }
I20260621 07:52:02.899449 16902 consensus_queue.cc:260] T d451d2f95842424a8f7ef18bcfb4cf54 P fc26fdec3ed34f8c8d9826274ea2b804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b977f65b3b44bfcb5bbc7a1edb102fd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 36113 } } peers { permanent_uuid: "87598af7bb284c969bdc9f8b94bb0044" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33777 } } peers { permanent_uuid: "fc26fdec3ed34f8c8d9826274ea2b804" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39747 } }
I20260621 07:52:02.900022 16751 heartbeater.cc:503] Master 127.12.12.62:37297 was elected leader, sending a full tablet report...
I20260621 07:52:02.900091 16901 ts_tablet_manager.cc:1428] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:52:02.901078 16618 heartbeater.cc:503] Master 127.12.12.62:37297 was elected leader, sending a full tablet report...
I20260621 07:52:02.902545 16902 ts_tablet_manager.cc:1428] T d451d2f95842424a8f7ef18bcfb4cf54 P fc26fdec3ed34f8c8d9826274ea2b804: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260621 07:52:03.066684 16885 tablet.cc:2369] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:52:03.080670 16907 raft_consensus.cc:482] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:52:03.080914 16907 raft_consensus.cc:504] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b977f65b3b44bfcb5bbc7a1edb102fd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 36113 } } peers { permanent_uuid: "87598af7bb284c969bdc9f8b94bb0044" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33777 } } peers { permanent_uuid: "fc26fdec3ed34f8c8d9826274ea2b804" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39747 } }
I20260621 07:52:03.082188 16907 leader_election.cc:288] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b977f65b3b44bfcb5bbc7a1edb102fd (127.12.12.2:36113), fc26fdec3ed34f8c8d9826274ea2b804 (127.12.12.1:39747)
I20260621 07:52:03.086570 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.082614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:36397 (local address 127.12.12.1:39747)
0621 07:52:03.082762 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:03.082766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:03.082780 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:52:03.082921 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:03.082925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:03.082984 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:03.083087 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:03.083614 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:03.084228 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:03.085493 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:03.085932 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:03.086015 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:03.086108 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:03.086313 (+ 205us) server_negotiation.cc:299] Negotiation successful
0621 07:52:03.086398 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:52:03.086587 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.082592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:35969 (local address 127.12.12.2:36113)
0621 07:52:03.082703 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:03.082708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:03.083160 (+ 452us) server_negotiation.cc:407] Connection header received
0621 07:52:03.083304 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:03.083308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:03.083368 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:03.083450 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:03.083884 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:03.084454 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:03.085564 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:03.085966 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:03.086097 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:03.086165 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:03.086371 (+ 206us) server_negotiation.cc:299] Negotiation successful
0621 07:52:03.086473 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 07:52:03.086587 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.082480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:39747 (local address 127.12.12.3:36397)
0621 07:52:03.082646 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:03.082661 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:03.082769 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:03.083102 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:03.083104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:03.083175 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:03.083325 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:03.083332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:03.084344 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:03.084348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:03.085355 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:03.085364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:03.086056 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:03.086063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:03.086166 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:03.086196 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 07:52:03.086267 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 07:52:03.086364 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":38,"spinlock_wait_cycles":34432}
I20260621 07:52:03.086598 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.082492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:36113 (local address 127.12.12.3:35969)
0621 07:52:03.083043 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:03.083060 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:03.083186 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:03.083613 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:03.083616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:03.083634 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:03.083750 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:03.083757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:03.084573 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:03.084575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:03.085404 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:03.085441 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:03.086120 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:03.086127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:03.086218 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:03.086234 (+ 16us) client_negotiation.cc:769] Sending connection context
0621 07:52:03.086331 (+ 97us) client_negotiation.cc:240] Negotiation successful
0621 07:52:03.086405 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":181,"spinlock_wait_cycles":9728,"thread_start_us":141,"threads_started":1}
I20260621 07:52:03.087283 16572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d451d2f95842424a8f7ef18bcfb4cf54" candidate_uuid: "87598af7bb284c969bdc9f8b94bb0044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc26fdec3ed34f8c8d9826274ea2b804" is_pre_election: true
I20260621 07:52:03.087357 16705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d451d2f95842424a8f7ef18bcfb4cf54" candidate_uuid: "87598af7bb284c969bdc9f8b94bb0044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b977f65b3b44bfcb5bbc7a1edb102fd" is_pre_election: true
I20260621 07:52:03.087630 16572 raft_consensus.cc:2454] T d451d2f95842424a8f7ef18bcfb4cf54 P fc26fdec3ed34f8c8d9826274ea2b804 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87598af7bb284c969bdc9f8b94bb0044 in term 0.
I20260621 07:52:03.087723 16705 raft_consensus.cc:2454] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87598af7bb284c969bdc9f8b94bb0044 in term 0.
I20260621 07:52:03.088248 16772 leader_election.cc:302] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4b977f65b3b44bfcb5bbc7a1edb102fd, 87598af7bb284c969bdc9f8b94bb0044, fc26fdec3ed34f8c8d9826274ea2b804; no voters:
I20260621 07:52:03.088560 16907 raft_consensus.cc:2781] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:52:03.088639 16907 raft_consensus.cc:482] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:52:03.088701 16907 raft_consensus.cc:3037] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:03.089926 16907 raft_consensus.cc:504] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b977f65b3b44bfcb5bbc7a1edb102fd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 36113 } } peers { permanent_uuid: "87598af7bb284c969bdc9f8b94bb0044" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33777 } } peers { permanent_uuid: "fc26fdec3ed34f8c8d9826274ea2b804" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39747 } }
I20260621 07:52:03.090380 16907 leader_election.cc:288] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [CANDIDATE]: Term 1 election: Requested vote from peers 4b977f65b3b44bfcb5bbc7a1edb102fd (127.12.12.2:36113), fc26fdec3ed34f8c8d9826274ea2b804 (127.12.12.1:39747)
W20260621 07:52:03.090631 16752 tablet.cc:2369] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:52:03.090808 16705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d451d2f95842424a8f7ef18bcfb4cf54" candidate_uuid: "87598af7bb284c969bdc9f8b94bb0044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b977f65b3b44bfcb5bbc7a1edb102fd"
I20260621 07:52:03.090939 16705 raft_consensus.cc:3037] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:03.091029 16572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d451d2f95842424a8f7ef18bcfb4cf54" candidate_uuid: "87598af7bb284c969bdc9f8b94bb0044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc26fdec3ed34f8c8d9826274ea2b804"
I20260621 07:52:03.091176 16572 raft_consensus.cc:3037] T d451d2f95842424a8f7ef18bcfb4cf54 P fc26fdec3ed34f8c8d9826274ea2b804 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:03.092077 16705 raft_consensus.cc:2454] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87598af7bb284c969bdc9f8b94bb0044 in term 1.
I20260621 07:52:03.092332 16572 raft_consensus.cc:2454] T d451d2f95842424a8f7ef18bcfb4cf54 P fc26fdec3ed34f8c8d9826274ea2b804 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87598af7bb284c969bdc9f8b94bb0044 in term 1.
I20260621 07:52:03.092439 16772 leader_election.cc:302] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b977f65b3b44bfcb5bbc7a1edb102fd, 87598af7bb284c969bdc9f8b94bb0044; no voters:
I20260621 07:52:03.092651 16907 raft_consensus.cc:2781] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:03.092878 16907 raft_consensus.cc:686] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [term 1 LEADER]: Becoming Leader. State: Replica: 87598af7bb284c969bdc9f8b94bb0044, State: Running, Role: LEADER
I20260621 07:52:03.093225 16907 consensus_queue.cc:237] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b977f65b3b44bfcb5bbc7a1edb102fd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 36113 } } peers { permanent_uuid: "87598af7bb284c969bdc9f8b94bb0044" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33777 } } peers { permanent_uuid: "fc26fdec3ed34f8c8d9826274ea2b804" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39747 } }
I20260621 07:52:03.096424 16428 catalog_manager.cc:5495] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87598af7bb284c969bdc9f8b94bb0044 (127.12.12.3). New cstate: current_term: 1 leader_uuid: "87598af7bb284c969bdc9f8b94bb0044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b977f65b3b44bfcb5bbc7a1edb102fd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 36113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87598af7bb284c969bdc9f8b94bb0044" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 33777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc26fdec3ed34f8c8d9826274ea2b804" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39747 } health_report { overall_health: UNKNOWN } } }
W20260621 07:52:03.114552 16619 tablet.cc:2369] T d451d2f95842424a8f7ef18bcfb4cf54 P fc26fdec3ed34f8c8d9826274ea2b804: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:52:03.115021 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.111214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46946 (local address 127.12.12.3:33777)
0621 07:52:03.111354 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:03.111358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:03.111372 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:52:03.111535 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:03.111538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:03.111602 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:03.111716 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:52:03.111959 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:03.112507 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:03.113388 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:03.113621 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:03.113727 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:03.114043 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:52:03.114608 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:52:03.114751 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:03.114830 (+ 79us) server_negotiation.cc:299] Negotiation successful
0621 07:52:03.114892 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:52:03.120875 16572 raft_consensus.cc:1261] T d451d2f95842424a8f7ef18bcfb4cf54 P fc26fdec3ed34f8c8d9826274ea2b804 [term 1 FOLLOWER]: Refusing update from remote peer 87598af7bb284c969bdc9f8b94bb0044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:03.120877 16705 raft_consensus.cc:1261] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd [term 1 FOLLOWER]: Refusing update from remote peer 87598af7bb284c969bdc9f8b94bb0044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:03.121588 16911 consensus_queue.cc:1035] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc26fdec3ed34f8c8d9826274ea2b804" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:03.121840 16907 consensus_queue.cc:1035] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b977f65b3b44bfcb5bbc7a1edb102fd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 36113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:03.132146 16915 mvcc.cc:205] Tried to move back new op lower bound from 7299188011495837696 to 7299188011392335872. Current Snapshot: MvccSnapshot[applied={T|T < 7299188011495837696}]
I20260621 07:52:03.138811 16426 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:03.139128 16426 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:03.141562 16426 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf1c81564004494fbba7b0976906d5eb: Sending DeleteTablet for 3 replicas of tablet d451d2f95842424a8f7ef18bcfb4cf54
I20260621 07:52:03.142432 16818 tablet_service.cc:1514] Processing DeleteTablet for tablet d451d2f95842424a8f7ef18bcfb4cf54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:03 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:46944
I20260621 07:52:03.142489 16685 tablet_service.cc:1514] Processing DeleteTablet for tablet d451d2f95842424a8f7ef18bcfb4cf54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:03 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:57804
I20260621 07:52:03.142511 16552 tablet_service.cc:1514] Processing DeleteTablet for tablet d451d2f95842424a8f7ef18bcfb4cf54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:03 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:38238
I20260621 07:52:03.142967 16925 tablet_replica.cc:330] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044: stopping tablet replica
I20260621 07:52:03.142975 16924 tablet_replica.cc:330] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd: stopping tablet replica
I20260621 07:52:03.143313 16924 raft_consensus.cc:2229] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:03.143316 16925 raft_consensus.cc:2229] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:52:03.143651 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 16490
I20260621 07:52:03.143673 16924 raft_consensus.cc:2258] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:03.145865 16925 raft_consensus.cc:2258] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:03.147078 16925 ts_tablet_manager.cc:1905] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:03.147081 16924 ts_tablet_manager.cc:1905] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:03.149828 16925 ts_tablet_manager.cc:1918] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:03.149838 16924 ts_tablet_manager.cc:1918] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:03.149941 16924 log.cc:1196] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/d451d2f95842424a8f7ef18bcfb4cf54
I20260621 07:52:03.149943 16925 log.cc:1196] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/d451d2f95842424a8f7ef18bcfb4cf54
I20260621 07:52:03.150290 16924 ts_tablet_manager.cc:1939] T d451d2f95842424a8f7ef18bcfb4cf54 P 4b977f65b3b44bfcb5bbc7a1edb102fd: Deleting consensus metadata
I20260621 07:52:03.150292 16925 ts_tablet_manager.cc:1939] T d451d2f95842424a8f7ef18bcfb4cf54 P 87598af7bb284c969bdc9f8b94bb0044: Deleting consensus metadata
I20260621 07:52:03.151772 16413 catalog_manager.cc:4841] TS 87598af7bb284c969bdc9f8b94bb0044 (127.12.12.3:33777): tablet d451d2f95842424a8f7ef18bcfb4cf54 (table test-table [id=889e275c127047548f43ccf2be71483b]) successfully deleted
I20260621 07:52:03.152016 16413 catalog_manager.cc:4841] TS 4b977f65b3b44bfcb5bbc7a1edb102fd (127.12.12.2:36113): tablet d451d2f95842424a8f7ef18bcfb4cf54 (table test-table [id=889e275c127047548f43ccf2be71483b]) successfully deleted
W20260621 07:52:03.155120 16415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.1:39747 (error 108)
I20260621 07:52:03.155169 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 16623
I20260621 07:52:03.158818 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.156701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:39747 (local address 127.0.0.1:38240)
0621 07:52:03.158594 (+ 1893us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:03.158694 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1772}
W20260621 07:52:03.159175 16415 catalog_manager.cc:4584] TS fc26fdec3ed34f8c8d9826274ea2b804 (127.12.12.1:39747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d451d2f95842424a8f7ef18bcfb4cf54: Network error: Client connection negotiation failed: client connection to 127.12.12.1:39747: connect: Connection refused (error 111)
I20260621 07:52:03.161597 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 16756
I20260621 07:52:03.166828 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 16397
2026-06-21T07:52:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2473 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-21T07:52:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:52:03Z Disabled control of system clock
I20260621 07:52:03.197079 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:38741
--webserver_interface=127.12.12.62
--webserver_port=0
--builtin_ntp_servers=127.12.12.20:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:38741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 07:52:03.312167 16931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:03.312542 16931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:03.312614 16931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:03.316810 16931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:03.316887 16931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:03.316902 16931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:03.316919 16931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:03.316936 16931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:52:03.321990 16931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:38741
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:38741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:03.323246 16931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:03.324704 16931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:03.333551 16936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:03.333619 16931 server_base.cc:1034] running on GCE node
W20260621 07:52:03.333633 16937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:03.333633 16939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:03.334511 16931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:03.335775 16931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:03.336974 16931 hybrid_clock.cc:648] HybridClock initialized: now 1782028323336961 us; error 33 us; skew 500 ppm
I20260621 07:52:03.339130 16931 webserver.cc:460] Webserver started at http://127.12.12.62:37341/ using document root <none> and password file <none>
I20260621 07:52:03.339772 16931 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:03.339871 16931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:03.340116 16931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:03.342254 16931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "05c8cc70d06e4928be2db73ccd6f6997"
format_stamp: "Formatted at 2026-06-21 07:52:03 on dist-test-slave-lbpt"
server_key: "6073253546ef97ff97e673555b08bdd1"
server_key_iv: "e02513e2e129387208c7d98c5265be4b"
server_key_version: "encryptionkey@0"
I20260621 07:52:03.342932 16931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "05c8cc70d06e4928be2db73ccd6f6997"
format_stamp: "Formatted at 2026-06-21 07:52:03 on dist-test-slave-lbpt"
server_key: "6073253546ef97ff97e673555b08bdd1"
server_key_iv: "e02513e2e129387208c7d98c5265be4b"
server_key_version: "encryptionkey@0"
I20260621 07:52:03.346776 16931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:52:03.349414 16945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:03.350833 16931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:03.350971 16931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "05c8cc70d06e4928be2db73ccd6f6997"
format_stamp: "Formatted at 2026-06-21 07:52:03 on dist-test-slave-lbpt"
server_key: "6073253546ef97ff97e673555b08bdd1"
server_key_iv: "e02513e2e129387208c7d98c5265be4b"
server_key_version: "encryptionkey@0"
I20260621 07:52:03.351095 16931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:03.355737 16931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:03.356336 16931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:03.356520 16931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:03.363366 16931 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:38741
I20260621 07:52:03.363384 16997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:38741 every 8 connection(s)
I20260621 07:52:03.364310 16931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:52:03.367671 16998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:03.373188 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 16931
I20260621 07:52:03.373329 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:52:03.373569 16998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997: Bootstrap starting.
I20260621 07:52:03.373625 12336 external_mini_cluster.cc:1428] Setting key 4a590f1f6cc5bdd5bdcc597f712297fb
I20260621 07:52:03.376287 16998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:03.377068 16998 log.cc:824] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:03.379136 16998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997: No bootstrap required, opened a new log
I20260621 07:52:03.379739 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.375068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60320 (local address 127.12.12.62:38741)
0621 07:52:03.375758 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:03.375776 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:03.375810 (+ 34us) server_negotiation.cc:407] Connection header received
0621 07:52:03.376542 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:03.376563 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:03.376925 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:03.377278 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:03.378289 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:03.378294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:03.378304 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:03.378312 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:03.378359 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:03.378374 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:03.378553 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:03.378703 (+ 150us) server_negotiation.cc:299] Negotiation successful
0621 07:52:03.378953 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":151,"threads_started":1}
I20260621 07:52:03.383141 16998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c8cc70d06e4928be2db73ccd6f6997" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38741 } }
I20260621 07:52:03.383389 16998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:03.383469 16998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05c8cc70d06e4928be2db73ccd6f6997, State: Initialized, Role: FOLLOWER
I20260621 07:52:03.383893 16998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c8cc70d06e4928be2db73ccd6f6997" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38741 } }
I20260621 07:52:03.384045 16998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:52:03.384131 16998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:52:03.384248 16998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:03.385191 16998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c8cc70d06e4928be2db73ccd6f6997" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38741 } }
I20260621 07:52:03.385576 16998 leader_election.cc:302] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05c8cc70d06e4928be2db73ccd6f6997; no voters:
I20260621 07:52:03.385958 16998 leader_election.cc:288] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:52:03.386091 17003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:03.386333 17003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [term 1 LEADER]: Becoming Leader. State: Replica: 05c8cc70d06e4928be2db73ccd6f6997, State: Running, Role: LEADER
I20260621 07:52:03.386677 17003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c8cc70d06e4928be2db73ccd6f6997" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38741 } }
I20260621 07:52:03.387292 16998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:52:03.388670 17004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05c8cc70d06e4928be2db73ccd6f6997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c8cc70d06e4928be2db73ccd6f6997" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38741 } } }
I20260621 07:52:03.388690 17005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05c8cc70d06e4928be2db73ccd6f6997. Latest consensus state: current_term: 1 leader_uuid: "05c8cc70d06e4928be2db73ccd6f6997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c8cc70d06e4928be2db73ccd6f6997" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38741 } } }
I20260621 07:52:03.388861 17005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:03.389133 17004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997 [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:03.389246 17012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:52:03.392519 17012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:52:03.398277 17012 catalog_manager.cc:1347] Generated new cluster ID: 92fe7d2c6cb14daeafd1400ccca11e07
I20260621 07:52:03.398461 17012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:52:03.412600 17012 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:52:03.412930 17012 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:52:03.418120 17012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997: Generated new TSK 0
I20260621 07:52:03.418936 17012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:52:03.422053 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38741
--builtin_ntp_servers=127.12.12.20:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 07:52:03.537119 17022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:03.537488 17022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:03.537532 17022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:03.541618 17022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:03.541687 17022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:03.541747 17022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:52:03.546571 17022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:03.547852 17022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:03.549829 17022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:03.560722 17028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:03.560699 17030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:03.560732 17027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:03.560772 17022 server_base.cc:1034] running on GCE node
I20260621 07:52:03.561444 17022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:03.562189 17022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:03.563369 17022 hybrid_clock.cc:648] HybridClock initialized: now 1782028323563344 us; error 34 us; skew 500 ppm
I20260621 07:52:03.565512 17022 webserver.cc:460] Webserver started at http://127.12.12.1:36291/ using document root <none> and password file <none>
I20260621 07:52:03.566124 17022 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:03.566187 17022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:03.566506 17022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:03.568678 17022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "59410b21f59a44eabb41ff791cb22391"
format_stamp: "Formatted at 2026-06-21 07:52:03 on dist-test-slave-lbpt"
server_key: "6465e9f25fc6033cd03b8ee3d0e6a65a"
server_key_iv: "3cc64f71967ac0940c04c621cbbc1ce7"
server_key_version: "encryptionkey@0"
I20260621 07:52:03.569267 17022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "59410b21f59a44eabb41ff791cb22391"
format_stamp: "Formatted at 2026-06-21 07:52:03 on dist-test-slave-lbpt"
server_key: "6465e9f25fc6033cd03b8ee3d0e6a65a"
server_key_iv: "3cc64f71967ac0940c04c621cbbc1ce7"
server_key_version: "encryptionkey@0"
I20260621 07:52:03.573240 17022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260621 07:52:03.576011 17036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:03.577412 17022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:52:03.577587 17022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "59410b21f59a44eabb41ff791cb22391"
format_stamp: "Formatted at 2026-06-21 07:52:03 on dist-test-slave-lbpt"
server_key: "6465e9f25fc6033cd03b8ee3d0e6a65a"
server_key_iv: "3cc64f71967ac0940c04c621cbbc1ce7"
server_key_version: "encryptionkey@0"
I20260621 07:52:03.577714 17022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:03.582624 17022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:03.583374 17022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:03.583626 17022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:03.584230 17022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:03.585223 17022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:03.585271 17022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:03.585343 17022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:03.585390 17022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:03.594300 17022 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:36769
I20260621 07:52:03.594363 17149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:36769 every 8 connection(s)
I20260621 07:52:03.595336 17022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:52:03.598483 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 17022
I20260621 07:52:03.598634 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:52:03.598964 12336 external_mini_cluster.cc:1428] Setting key 4e4fc3d875ec2916fa11a4c9facc8c70
I20260621 07:52:03.601152 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38741
--builtin_ntp_servers=127.12.12.20:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 07:52:03.602759 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.597331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:36645 (local address 127.12.12.62:38741)
0621 07:52:03.597503 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:03.597507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:03.598967 (+ 1460us) server_negotiation.cc:407] Connection header received
0621 07:52:03.600686 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:03.600689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:03.600724 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:03.600791 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:03.602133 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:03.602137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:03.602139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:03.602151 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:03.602171 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:03.602180 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:03.602266 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:03.602520 (+ 254us) server_negotiation.cc:299] Negotiation successful
0621 07:52:03.602583 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:52:03.603669 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.597826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38741 (local address 127.12.12.1:36645)
0621 07:52:03.598773 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:03.598837 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:03.600445 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:03.600994 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:03.601006 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:03.601135 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:03.601735 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:52:03.601740 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:03.601802 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:52:03.601897 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:03.602276 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:03.602288 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:03.602297 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 07:52:03.602534 (+ 237us) client_negotiation.cc:240] Negotiation successful
0621 07:52:03.602803 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":135,"threads_started":1}
I20260621 07:52:03.605095 17150 heartbeater.cc:347] Connected to a master server at 127.12.12.62:38741
I20260621 07:52:03.605335 17150 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:03.605783 17150 heartbeater.cc:511] Master 127.12.12.62:38741 requested a full tablet report, sending...
I20260621 07:52:03.607687 16962 ts_manager.cc:194] Registered new tserver with Master: 59410b21f59a44eabb41ff791cb22391 (127.12.12.1:36769)
W20260621 07:52:03.719424 17153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:03.719733 17153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:03.719776 17153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:03.723950 17153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:03.724031 17153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:03.724110 17153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:52:03.729176 17153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:03.730368 17153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:03.731927 17153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:03.742457 17161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:03.742466 17158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:03.742532 17153 server_base.cc:1034] running on GCE node
W20260621 07:52:03.742466 17159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:03.743101 17153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:03.743819 17153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:03.744994 17153 hybrid_clock.cc:648] HybridClock initialized: now 1782028323744978 us; error 29 us; skew 500 ppm
I20260621 07:52:03.746951 17153 webserver.cc:460] Webserver started at http://127.12.12.2:46749/ using document root <none> and password file <none>
I20260621 07:52:03.747538 17153 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:03.747605 17153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:03.747781 17153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:03.750025 17153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "5a8bab11ca644f4d861f8471cfb7eee8"
format_stamp: "Formatted at 2026-06-21 07:52:03 on dist-test-slave-lbpt"
server_key: "5a375089e2230f42be683d41582ea21e"
server_key_iv: "71d041f140f2c09e85c2d4e4703365e1"
server_key_version: "encryptionkey@0"
I20260621 07:52:03.750659 17153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "5a8bab11ca644f4d861f8471cfb7eee8"
format_stamp: "Formatted at 2026-06-21 07:52:03 on dist-test-slave-lbpt"
server_key: "5a375089e2230f42be683d41582ea21e"
server_key_iv: "71d041f140f2c09e85c2d4e4703365e1"
server_key_version: "encryptionkey@0"
I20260621 07:52:03.754403 17153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:52:03.757330 17167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:03.758697 17153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:03.758795 17153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "5a8bab11ca644f4d861f8471cfb7eee8"
format_stamp: "Formatted at 2026-06-21 07:52:03 on dist-test-slave-lbpt"
server_key: "5a375089e2230f42be683d41582ea21e"
server_key_iv: "71d041f140f2c09e85c2d4e4703365e1"
server_key_version: "encryptionkey@0"
I20260621 07:52:03.758891 17153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:03.763455 17153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:03.764048 17153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:03.764190 17153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:03.764695 17153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:03.765579 17153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:03.765625 17153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:03.765666 17153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:03.765681 17153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:03.775270 17153 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:46281
I20260621 07:52:03.775288 17280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:46281 every 8 connection(s)
I20260621 07:52:03.776257 17153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:52:03.781843 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 17153
I20260621 07:52:03.782027 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:52:03.782470 12336 external_mini_cluster.cc:1428] Setting key 701d7aa3c80925689442176b72048834
I20260621 07:52:03.783458 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.778507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:54433 (local address 127.12.12.62:38741)
0621 07:52:03.778738 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:03.778742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:03.779892 (+ 1150us) server_negotiation.cc:407] Connection header received
0621 07:52:03.781361 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:03.781365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:03.781415 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:03.781501 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:03.782950 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:03.782960 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:03.782962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:03.782975 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:03.782997 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:03.783005 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:03.783131 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:03.783278 (+ 147us) server_negotiation.cc:299] Negotiation successful
0621 07:52:03.783328 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:52:03.784495 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.778918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38741 (local address 127.12.12.2:54433)
0621 07:52:03.779725 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:03.779771 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:03.781143 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:03.781742 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:03.781750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:03.781847 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:03.782664 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:52:03.782666 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:03.782709 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:52:03.782781 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:03.783116 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:03.783125 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:03.783131 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:52:03.783289 (+ 158us) client_negotiation.cc:240] Negotiation successful
0621 07:52:03.783559 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":141,"threads_started":1}
I20260621 07:52:03.784807 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38741
--builtin_ntp_servers=127.12.12.20:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 07:52:03.785665 17281 heartbeater.cc:347] Connected to a master server at 127.12.12.62:38741
I20260621 07:52:03.785935 17281 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:03.786515 17281 heartbeater.cc:511] Master 127.12.12.62:38741 requested a full tablet report, sending...
I20260621 07:52:03.787645 16962 ts_manager.cc:194] Registered new tserver with Master: 5a8bab11ca644f4d861f8471cfb7eee8 (127.12.12.2:46281)
W20260621 07:52:03.902725 17284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:03.903061 17284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:03.903138 17284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:03.907393 17284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:03.907502 17284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:03.907634 17284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:52:03.912683 17284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:03.913972 17284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:03.915441 17284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:03.925287 17289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:03.925297 17290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:03.925357 17284 server_base.cc:1034] running on GCE node
W20260621 07:52:03.925309 17292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:03.925900 17284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:03.926606 17284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:03.927800 17284 hybrid_clock.cc:648] HybridClock initialized: now 1782028323927770 us; error 42 us; skew 500 ppm
I20260621 07:52:03.929808 17284 webserver.cc:460] Webserver started at http://127.12.12.3:34135/ using document root <none> and password file <none>
I20260621 07:52:03.930500 17284 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:03.930569 17284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:03.930794 17284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:03.932914 17284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "a4d51d4e599943a3928d6ff1ed09c824"
format_stamp: "Formatted at 2026-06-21 07:52:03 on dist-test-slave-lbpt"
server_key: "2f5922eaa81c13c8efa124872efb9826"
server_key_iv: "b4ce2c9ebb81bc5ba317d4e899dbb26d"
server_key_version: "encryptionkey@0"
I20260621 07:52:03.933487 17284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "a4d51d4e599943a3928d6ff1ed09c824"
format_stamp: "Formatted at 2026-06-21 07:52:03 on dist-test-slave-lbpt"
server_key: "2f5922eaa81c13c8efa124872efb9826"
server_key_iv: "b4ce2c9ebb81bc5ba317d4e899dbb26d"
server_key_version: "encryptionkey@0"
I20260621 07:52:03.937274 17284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260621 07:52:03.939909 17298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:03.941255 17284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260621 07:52:03.941354 17284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "a4d51d4e599943a3928d6ff1ed09c824"
format_stamp: "Formatted at 2026-06-21 07:52:03 on dist-test-slave-lbpt"
server_key: "2f5922eaa81c13c8efa124872efb9826"
server_key_iv: "b4ce2c9ebb81bc5ba317d4e899dbb26d"
server_key_version: "encryptionkey@0"
I20260621 07:52:03.941578 17284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:03.946074 17284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:03.946719 17284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:03.946897 17284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:03.947425 17284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:03.948318 17284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:03.948364 17284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:03.948429 17284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:03.948477 17284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:03.957381 17284 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:35771
I20260621 07:52:03.957525 17411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:35771 every 8 connection(s)
I20260621 07:52:03.958384 17284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:52:03.962014 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 17284
I20260621 07:52:03.962114 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:52:03.962337 12336 external_mini_cluster.cc:1428] Setting key 057308c0823639e2c58b0ead04d1b20c
I20260621 07:52:03.966046 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.960385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:43607 (local address 127.12.12.62:38741)
0621 07:52:03.960540 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:03.960544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:03.961698 (+ 1154us) server_negotiation.cc:407] Connection header received
0621 07:52:03.964240 (+ 2542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:03.964242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:03.964281 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:03.964359 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:03.965638 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:03.965641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:03.965643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:03.965654 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:03.965675 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:03.965683 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:03.965773 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:03.965899 (+ 126us) server_negotiation.cc:299] Negotiation successful
0621 07:52:03.965937 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 07:52:03.967105 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.960812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38741 (local address 127.12.12.3:43607)
0621 07:52:03.961531 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:03.961575 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:03.963961 (+ 2386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:03.964583 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:03.964593 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:03.964711 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:03.965325 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:52:03.965327 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:03.965369 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:52:03.965439 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:03.965771 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:03.965777 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:03.965782 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 07:52:03.965936 (+ 154us) client_negotiation.cc:240] Negotiation successful
0621 07:52:03.966243 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":115,"threads_started":1}
I20260621 07:52:03.968466 17412 heartbeater.cc:347] Connected to a master server at 127.12.12.62:38741
I20260621 07:52:03.968731 17412 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:03.969170 17412 heartbeater.cc:511] Master 127.12.12.62:38741 requested a full tablet report, sending...
I20260621 07:52:03.970153 16962 ts_manager.cc:194] Registered new tserver with Master: a4d51d4e599943a3928d6ff1ed09c824 (127.12.12.3:35771)
I20260621 07:52:03.976847 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:52:03.980805 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.979062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.12.12.62:38741)
0621 07:52:03.979249 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:03.979254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:03.979402 (+ 148us) server_negotiation.cc:407] Connection header received
0621 07:52:03.979575 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:03.979578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:03.979616 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:03.979707 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:03.980369 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:03.980373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:03.980376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:03.980388 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:03.980412 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:03.980425 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:03.980511 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:03.980616 (+ 105us) server_negotiation.cc:299] Negotiation successful
0621 07:52:03.980663 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260621 07:52:03.983851 16962 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:52:03.986948 16962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:52:04.001209 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.997578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:46281 (local address 127.0.0.1:58340)
0621 07:52:03.998718 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:03.998732 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:03.998815 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:03.999348 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:03.999356 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:03.999434 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:04.000235 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:52:04.000236 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:04.000291 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:52:04.000369 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:04.000812 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:04.000823 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:04.000829 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:52:04.000969 (+ 140us) client_negotiation.cc:240] Negotiation successful
0621 07:52:04.001048 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"thread_start_us":112,"threads_started":1}
I20260621 07:52:04.001675 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.998157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58340 (local address 127.12.12.2:46281)
0621 07:52:03.998842 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:03.998855 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:03.998881 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:52:03.998936 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:03.998942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:03.999164 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:03.999330 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:04.000564 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:04.000570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:04.000583 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:04.000597 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:04.000648 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:04.000691 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:04.000799 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:04.001424 (+ 625us) server_negotiation.cc:299] Negotiation successful
0621 07:52:04.001509 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":135,"threads_started":1}
I20260621 07:52:04.003484 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.997479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:36769 (local address 127.0.0.1:52820)
0621 07:52:03.998384 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:03.998462 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:03.998574 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:04.001507 (+ 2933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:04.001511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:04.001526 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:04.002669 (+ 1143us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:52:04.002671 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:04.002685 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:52:04.002717 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:04.003239 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:04.003244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:04.003247 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 07:52:04.003303 (+ 56us) client_negotiation.cc:240] Negotiation successful
0621 07:52:04.003338 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":94,"threads_started":1}
I20260621 07:52:04.003674 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.998232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:35771 (local address 127.0.0.1:32998)
0621 07:52:04.000879 (+ 2647us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:04.000895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:04.000984 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:04.002166 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:04.002170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:04.002184 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:04.002801 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:52:04.002803 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:04.002814 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:52:04.002841 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:04.003409 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:04.003413 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:04.003416 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 07:52:04.003499 (+ 83us) client_negotiation.cc:240] Negotiation successful
0621 07:52:04.003532 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2579,"thread_start_us":80,"threads_started":1}
I20260621 07:52:04.004057 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.998651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.12.12.3:35771)
0621 07:52:03.999097 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:03.999107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:04.001094 (+ 1987us) server_negotiation.cc:407] Connection header received
0621 07:52:04.001155 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:04.001160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:04.001339 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:04.001470 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:04.003212 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:04.003221 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:04.003235 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:04.003247 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:04.003291 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:04.003326 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:04.003424 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:04.003863 (+ 439us) server_negotiation.cc:299] Negotiation successful
0621 07:52:04.003942 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":133,"threads_started":1}
I20260621 07:52:04.004138 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:03.997663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52820 (local address 127.12.12.1:36769)
0621 07:52:04.001125 (+ 3462us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:04.001136 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:04.001156 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:52:04.001200 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:04.001204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:04.001338 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:04.001762 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:04.003015 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:04.003021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:04.003034 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:04.003046 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:04.003092 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:04.003129 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:04.003876 (+ 747us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:04.003926 (+ 50us) server_negotiation.cc:299] Negotiation successful
0621 07:52:04.003995 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3354,"thread_start_us":2932,"threads_started":1}
I20260621 07:52:04.005720 17215 tablet_service.cc:1467] Processing CreateTablet for tablet 162bf12cae8d4256a2ce016b8ed2a2f3 (DEFAULT_TABLE table=test-table [id=c7ba8c6d232f4b4bb0fb7e8a7d6f8280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:04.006808 17215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162bf12cae8d4256a2ce016b8ed2a2f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:04.006476 17346 tablet_service.cc:1467] Processing CreateTablet for tablet 162bf12cae8d4256a2ce016b8ed2a2f3 (DEFAULT_TABLE table=test-table [id=c7ba8c6d232f4b4bb0fb7e8a7d6f8280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:04.007504 17346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162bf12cae8d4256a2ce016b8ed2a2f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:04.013401 17429 tablet_bootstrap.cc:492] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824: Bootstrap starting.
I20260621 07:52:04.014210 17084 tablet_service.cc:1467] Processing CreateTablet for tablet 162bf12cae8d4256a2ce016b8ed2a2f3 (DEFAULT_TABLE table=test-table [id=c7ba8c6d232f4b4bb0fb7e8a7d6f8280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:04.014707 17430 tablet_bootstrap.cc:492] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8: Bootstrap starting.
I20260621 07:52:04.015391 17084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162bf12cae8d4256a2ce016b8ed2a2f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:04.015784 17429 tablet_bootstrap.cc:654] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:04.016525 17429 log.cc:824] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:04.016853 17430 tablet_bootstrap.cc:654] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:04.017705 17430 log.cc:824] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:04.018774 17429 tablet_bootstrap.cc:492] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824: No bootstrap required, opened a new log
I20260621 07:52:04.018988 17429 ts_tablet_manager.cc:1397] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:52:04.020200 17430 tablet_bootstrap.cc:492] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8: No bootstrap required, opened a new log
I20260621 07:52:04.020392 17430 ts_tablet_manager.cc:1397] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:52:04.022481 17429 raft_consensus.cc:348] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59410b21f59a44eabb41ff791cb22391" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36769 } } peers { permanent_uuid: "a4d51d4e599943a3928d6ff1ed09c824" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35771 } } peers { permanent_uuid: "5a8bab11ca644f4d861f8471cfb7eee8" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46281 } }
I20260621 07:52:04.022763 17429 raft_consensus.cc:374] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:04.022836 17429 raft_consensus.cc:729] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d51d4e599943a3928d6ff1ed09c824, State: Initialized, Role: FOLLOWER
I20260621 07:52:04.023293 17429 consensus_queue.cc:260] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59410b21f59a44eabb41ff791cb22391" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36769 } } peers { permanent_uuid: "a4d51d4e599943a3928d6ff1ed09c824" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35771 } } peers { permanent_uuid: "5a8bab11ca644f4d861f8471cfb7eee8" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46281 } }
I20260621 07:52:04.023945 17412 heartbeater.cc:503] Master 127.12.12.62:38741 was elected leader, sending a full tablet report...
I20260621 07:52:04.024322 17429 ts_tablet_manager.cc:1428] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:52:04.024402 17433 tablet_bootstrap.cc:492] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 59410b21f59a44eabb41ff791cb22391: Bootstrap starting.
I20260621 07:52:04.025192 17430 raft_consensus.cc:348] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59410b21f59a44eabb41ff791cb22391" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36769 } } peers { permanent_uuid: "a4d51d4e599943a3928d6ff1ed09c824" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35771 } } peers { permanent_uuid: "5a8bab11ca644f4d861f8471cfb7eee8" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46281 } }
I20260621 07:52:04.025488 17430 raft_consensus.cc:374] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:04.025537 17430 raft_consensus.cc:729] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8bab11ca644f4d861f8471cfb7eee8, State: Initialized, Role: FOLLOWER
I20260621 07:52:04.026007 17430 consensus_queue.cc:260] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59410b21f59a44eabb41ff791cb22391" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36769 } } peers { permanent_uuid: "a4d51d4e599943a3928d6ff1ed09c824" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35771 } } peers { permanent_uuid: "5a8bab11ca644f4d861f8471cfb7eee8" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46281 } }
I20260621 07:52:04.026724 17281 heartbeater.cc:503] Master 127.12.12.62:38741 was elected leader, sending a full tablet report...
I20260621 07:52:04.026872 17433 tablet_bootstrap.cc:654] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 59410b21f59a44eabb41ff791cb22391: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:04.027781 17433 log.cc:824] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 59410b21f59a44eabb41ff791cb22391: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:04.028003 17430 ts_tablet_manager.cc:1428] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260621 07:52:04.030156 17433 tablet_bootstrap.cc:492] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 59410b21f59a44eabb41ff791cb22391: No bootstrap required, opened a new log
I20260621 07:52:04.030400 17433 ts_tablet_manager.cc:1397] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 59410b21f59a44eabb41ff791cb22391: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:52:04.033957 17433 raft_consensus.cc:348] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 59410b21f59a44eabb41ff791cb22391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59410b21f59a44eabb41ff791cb22391" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36769 } } peers { permanent_uuid: "a4d51d4e599943a3928d6ff1ed09c824" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35771 } } peers { permanent_uuid: "5a8bab11ca644f4d861f8471cfb7eee8" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46281 } }
I20260621 07:52:04.034228 17433 raft_consensus.cc:374] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 59410b21f59a44eabb41ff791cb22391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:04.034305 17433 raft_consensus.cc:729] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 59410b21f59a44eabb41ff791cb22391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59410b21f59a44eabb41ff791cb22391, State: Initialized, Role: FOLLOWER
I20260621 07:52:04.034775 17433 consensus_queue.cc:260] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 59410b21f59a44eabb41ff791cb22391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59410b21f59a44eabb41ff791cb22391" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36769 } } peers { permanent_uuid: "a4d51d4e599943a3928d6ff1ed09c824" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35771 } } peers { permanent_uuid: "5a8bab11ca644f4d861f8471cfb7eee8" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46281 } }
I20260621 07:52:04.035485 17150 heartbeater.cc:503] Master 127.12.12.62:38741 was elected leader, sending a full tablet report...
I20260621 07:52:04.035785 17433 ts_tablet_manager.cc:1428] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 59410b21f59a44eabb41ff791cb22391: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
W20260621 07:52:04.096570 17151 tablet.cc:2369] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 59410b21f59a44eabb41ff791cb22391: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:52:04.209403 17413 tablet.cc:2369] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:52:04.277892 17282 tablet.cc:2369] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:52:04.354625 17434 raft_consensus.cc:482] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:52:04.354827 17434 raft_consensus.cc:504] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59410b21f59a44eabb41ff791cb22391" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36769 } } peers { permanent_uuid: "a4d51d4e599943a3928d6ff1ed09c824" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35771 } } peers { permanent_uuid: "5a8bab11ca644f4d861f8471cfb7eee8" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46281 } }
I20260621 07:52:04.356180 17434 leader_election.cc:288] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59410b21f59a44eabb41ff791cb22391 (127.12.12.1:36769), 5a8bab11ca644f4d861f8471cfb7eee8 (127.12.12.2:46281)
I20260621 07:52:04.358261 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:04.356387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:36769 (local address 127.12.12.3:47963)
0621 07:52:04.356527 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:04.356544 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:04.356634 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:04.357002 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:04.357005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:04.357024 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:04.357632 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:52:04.357635 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:04.357648 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:52:04.357681 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:04.357987 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:04.357992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:04.357997 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 07:52:04.358078 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 07:52:04.358115 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 07:52:04.358304 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:04.356606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:47963 (local address 127.12.12.1:36769)
0621 07:52:04.356731 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:04.356737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:04.356753 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:52:04.356811 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:04.356813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:04.356847 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:04.356958 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:04.357803 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:04.357810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:04.357815 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:04.357827 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:04.357849 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:04.357860 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:04.357944 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:04.358122 (+ 178us) server_negotiation.cc:299] Negotiation successful
0621 07:52:04.358171 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:52:04.358343 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:04.356630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:36699 (local address 127.12.12.2:46281)
0621 07:52:04.356738 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:04.356743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:04.356918 (+ 175us) server_negotiation.cc:407] Connection header received
0621 07:52:04.357061 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:04.357065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:04.357106 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:04.357161 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:04.357985 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:04.357988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:04.357991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:04.358004 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:04.358028 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:04.358036 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:04.358142 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:04.358200 (+ 58us) server_negotiation.cc:299] Negotiation successful
0621 07:52:04.358234 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:52:04.358374 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:04.356468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:46281 (local address 127.12.12.3:36699)
0621 07:52:04.356825 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:04.356835 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:04.356921 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:04.357318 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:04.357323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:04.357338 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:04.357815 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:52:04.357818 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:04.357830 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:52:04.357856 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:04.358124 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:04.358127 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:04.358130 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 07:52:04.358201 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 07:52:04.358226 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":130,"threads_started":1}
I20260621 07:52:04.359162 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162bf12cae8d4256a2ce016b8ed2a2f3" candidate_uuid: "a4d51d4e599943a3928d6ff1ed09c824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59410b21f59a44eabb41ff791cb22391" is_pre_election: true
I20260621 07:52:04.359182 17235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162bf12cae8d4256a2ce016b8ed2a2f3" candidate_uuid: "a4d51d4e599943a3928d6ff1ed09c824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8bab11ca644f4d861f8471cfb7eee8" is_pre_election: true
I20260621 07:52:04.359494 17104 raft_consensus.cc:2454] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 59410b21f59a44eabb41ff791cb22391 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4d51d4e599943a3928d6ff1ed09c824 in term 0.
I20260621 07:52:04.359491 17235 raft_consensus.cc:2454] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4d51d4e599943a3928d6ff1ed09c824 in term 0.
I20260621 07:52:04.360004 17299 leader_election.cc:302] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 59410b21f59a44eabb41ff791cb22391, 5a8bab11ca644f4d861f8471cfb7eee8, a4d51d4e599943a3928d6ff1ed09c824; no voters:
I20260621 07:52:04.360321 17434 raft_consensus.cc:2781] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:52:04.360421 17434 raft_consensus.cc:482] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:52:04.360545 17434 raft_consensus.cc:3037] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:04.361603 17434 raft_consensus.cc:504] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59410b21f59a44eabb41ff791cb22391" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36769 } } peers { permanent_uuid: "a4d51d4e599943a3928d6ff1ed09c824" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35771 } } peers { permanent_uuid: "5a8bab11ca644f4d861f8471cfb7eee8" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46281 } }
I20260621 07:52:04.362053 17434 leader_election.cc:288] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [CANDIDATE]: Term 1 election: Requested vote from peers 59410b21f59a44eabb41ff791cb22391 (127.12.12.1:36769), 5a8bab11ca644f4d861f8471cfb7eee8 (127.12.12.2:46281)
I20260621 07:52:04.362452 17235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162bf12cae8d4256a2ce016b8ed2a2f3" candidate_uuid: "a4d51d4e599943a3928d6ff1ed09c824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8bab11ca644f4d861f8471cfb7eee8"
I20260621 07:52:04.362571 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162bf12cae8d4256a2ce016b8ed2a2f3" candidate_uuid: "a4d51d4e599943a3928d6ff1ed09c824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59410b21f59a44eabb41ff791cb22391"
I20260621 07:52:04.362605 17235 raft_consensus.cc:3037] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:04.362707 17104 raft_consensus.cc:3037] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 59410b21f59a44eabb41ff791cb22391 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:04.363920 17104 raft_consensus.cc:2454] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 59410b21f59a44eabb41ff791cb22391 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4d51d4e599943a3928d6ff1ed09c824 in term 1.
I20260621 07:52:04.363936 17235 raft_consensus.cc:2454] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4d51d4e599943a3928d6ff1ed09c824 in term 1.
I20260621 07:52:04.364200 17299 leader_election.cc:302] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59410b21f59a44eabb41ff791cb22391, a4d51d4e599943a3928d6ff1ed09c824; no voters:
I20260621 07:52:04.364434 17434 raft_consensus.cc:2781] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:04.364665 17434 raft_consensus.cc:686] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [term 1 LEADER]: Becoming Leader. State: Replica: a4d51d4e599943a3928d6ff1ed09c824, State: Running, Role: LEADER
I20260621 07:52:04.365015 17434 consensus_queue.cc:237] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59410b21f59a44eabb41ff791cb22391" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36769 } } peers { permanent_uuid: "a4d51d4e599943a3928d6ff1ed09c824" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35771 } } peers { permanent_uuid: "5a8bab11ca644f4d861f8471cfb7eee8" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46281 } }
I20260621 07:52:04.368244 16962 catalog_manager.cc:5495] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4d51d4e599943a3928d6ff1ed09c824 (127.12.12.3). New cstate: current_term: 1 leader_uuid: "a4d51d4e599943a3928d6ff1ed09c824" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59410b21f59a44eabb41ff791cb22391" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4d51d4e599943a3928d6ff1ed09c824" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a8bab11ca644f4d861f8471cfb7eee8" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46281 } health_report { overall_health: UNKNOWN } } }
I20260621 07:52:04.452605 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:04.451017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33010 (local address 127.12.12.3:35771)
0621 07:52:04.451151 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:04.451156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:04.451171 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:52:04.451293 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:04.451295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:04.451358 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:04.451450 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:04.452197 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:04.452200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:04.452203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:04.452214 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:04.452236 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:04.452250 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:04.452343 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:04.452418 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 07:52:04.452457 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:52:04.458813 17104 raft_consensus.cc:1261] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 59410b21f59a44eabb41ff791cb22391 [term 1 FOLLOWER]: Refusing update from remote peer a4d51d4e599943a3928d6ff1ed09c824: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:04.458823 17235 raft_consensus.cc:1261] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8 [term 1 FOLLOWER]: Refusing update from remote peer a4d51d4e599943a3928d6ff1ed09c824: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:04.459615 17439 consensus_queue.cc:1035] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a8bab11ca644f4d861f8471cfb7eee8" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:04.459832 17434 consensus_queue.cc:1035] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59410b21f59a44eabb41ff791cb22391" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:04.478143 16962 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:04.478307 16962 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:04.480906 16962 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05c8cc70d06e4928be2db73ccd6f6997: Sending DeleteTablet for 3 replicas of tablet 162bf12cae8d4256a2ce016b8ed2a2f3
I20260621 07:52:04.481859 17215 tablet_service.cc:1514] Processing DeleteTablet for tablet 162bf12cae8d4256a2ce016b8ed2a2f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:04 UTC) from {username='slave'} at 127.0.0.1:58340
I20260621 07:52:04.481993 17084 tablet_service.cc:1514] Processing DeleteTablet for tablet 162bf12cae8d4256a2ce016b8ed2a2f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:04 UTC) from {username='slave'} at 127.0.0.1:52820
I20260621 07:52:04.481978 17346 tablet_service.cc:1514] Processing DeleteTablet for tablet 162bf12cae8d4256a2ce016b8ed2a2f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:04 UTC) from {username='slave'} at 127.0.0.1:32998
I20260621 07:52:04.482585 17454 tablet_replica.cc:330] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824: stopping tablet replica
I20260621 07:52:04.482908 17454 raft_consensus.cc:2229] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:52:04.483165 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 17022
I20260621 07:52:04.484793 17454 raft_consensus.cc:2258] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:04.485230 17453 tablet_replica.cc:330] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8: stopping tablet replica
I20260621 07:52:04.485569 17453 raft_consensus.cc:2229] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:04.485922 17453 raft_consensus.cc:2258] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:04.486886 17454 ts_tablet_manager.cc:1905] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:04.487614 17453 ts_tablet_manager.cc:1905] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:04.490067 17454 ts_tablet_manager.cc:1918] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:04.490264 17454 log.cc:1196] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/162bf12cae8d4256a2ce016b8ed2a2f3
W20260621 07:52:04.490948 16946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.1:36769 (error 108)
I20260621 07:52:04.491060 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 17153
I20260621 07:52:04.491250 17453 ts_tablet_manager.cc:1918] T 162bf12cae8d4256a2ce016b8ed2a2f3 P 5a8bab11ca644f4d861f8471cfb7eee8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:04.491888 17454 ts_tablet_manager.cc:1939] T 162bf12cae8d4256a2ce016b8ed2a2f3 P a4d51d4e599943a3928d6ff1ed09c824: Deleting consensus metadata
I20260621 07:52:04.493523 16946 catalog_manager.cc:4841] TS a4d51d4e599943a3928d6ff1ed09c824 (127.12.12.3:35771): tablet 162bf12cae8d4256a2ce016b8ed2a2f3 (table test-table [id=c7ba8c6d232f4b4bb0fb7e8a7d6f8280]) successfully deleted
I20260621 07:52:04.494319 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:04.493916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:36769 (local address 127.0.0.1:52826)
0621 07:52:04.494129 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:04.494211 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260621 07:52:04.494530 16946 catalog_manager.cc:4584] TS 59410b21f59a44eabb41ff791cb22391 (127.12.12.1:36769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 162bf12cae8d4256a2ce016b8ed2a2f3: Network error: Client connection negotiation failed: client connection to 127.12.12.1:36769: connect: Connection refused (error 111)
I20260621 07:52:04.497805 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 17284
I20260621 07:52:04.498870 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:04.497920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:46281 (local address 127.0.0.1:58352)
0621 07:52:04.498751 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:04.498799 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.2:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":756}
I20260621 07:52:04.503702 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 16931
2026-06-21T07:52:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1336 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260621 07:52:04.525413 12336 mini_oidc.cc:152] Starting JWKS server
I20260621 07:52:04.527249 12336 webserver.cc:460] Webserver started at https://127.0.0.1:46043/ using document root <none> and password file <none>
I20260621 07:52:04.527457 12336 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 07:52:04.527755 12336 webserver.cc:460] Webserver started at http://127.0.0.1:37183/ using document root <none> and password file <none>
2026-06-21T07:52:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:52:04Z Disabled control of system clock
I20260621 07:52:04.537256 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:42223
--webserver_interface=127.12.12.62
--webserver_port=0
--builtin_ntp_servers=127.12.12.20:42491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:42223
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46043/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/testchainca.cert with env {}
W20260621 07:52:04.653064 17461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:04.653360 17461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:04.653402 17461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:04.657508 17461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:04.657577 17461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:04.657593 17461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:04.657610 17461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:04.657625 17461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 07:52:04.657642 17461 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 07:52:04.657655 17461 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46043/jwks.json
I20260621 07:52:04.663305 17461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:42223
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:42223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46043/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:04.664392 17461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:04.665407 17461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:04.674265 17467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:04.674260 17469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:04.674665 17461 server_base.cc:1034] running on GCE node
W20260621 07:52:04.674260 17466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:04.675318 17461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:04.676448 17461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:04.677663 17461 hybrid_clock.cc:648] HybridClock initialized: now 1782028324677647 us; error 31 us; skew 500 ppm
I20260621 07:52:04.679797 17461 webserver.cc:460] Webserver started at http://127.12.12.62:42335/ using document root <none> and password file <none>
I20260621 07:52:04.680424 17461 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:04.680522 17461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:04.680760 17461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:04.682958 17461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "a064cce384494fd2a2766f6f9cd2bbe6"
format_stamp: "Formatted at 2026-06-21 07:52:04 on dist-test-slave-lbpt"
server_key: "49fb18892817366e23813859163f8e73"
server_key_iv: "0868ac968477da87d1c5a10ad5324b83"
server_key_version: "encryptionkey@0"
I20260621 07:52:04.683539 17461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "a064cce384494fd2a2766f6f9cd2bbe6"
format_stamp: "Formatted at 2026-06-21 07:52:04 on dist-test-slave-lbpt"
server_key: "49fb18892817366e23813859163f8e73"
server_key_iv: "0868ac968477da87d1c5a10ad5324b83"
server_key_version: "encryptionkey@0"
I20260621 07:52:04.687379 17461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:52:04.689926 17475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:04.691373 17461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:04.691529 17461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "a064cce384494fd2a2766f6f9cd2bbe6"
format_stamp: "Formatted at 2026-06-21 07:52:04 on dist-test-slave-lbpt"
server_key: "49fb18892817366e23813859163f8e73"
server_key_iv: "0868ac968477da87d1c5a10ad5324b83"
server_key_version: "encryptionkey@0"
I20260621 07:52:04.691692 17461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:04.707787 17461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:04.708565 17461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:04.708772 17461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:04.715852 17461 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:42223
I20260621 07:52:04.715888 17527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:42223 every 8 connection(s)
I20260621 07:52:04.716852 17461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:52:04.720258 17528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:04.723630 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 17461
I20260621 07:52:04.723757 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:52:04.724067 12336 external_mini_cluster.cc:1428] Setting key 63d132a3023d1c4409ab12733c15a459
I20260621 07:52:04.726593 17528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6: Bootstrap starting.
I20260621 07:52:04.729017 17528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:04.729996 17528 log.cc:824] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:04.732239 17528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6: No bootstrap required, opened a new log
I20260621 07:52:04.733846 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:04.725569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.12.12.62:42223)
0621 07:52:04.726258 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:04.726274 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:04.726309 (+ 35us) server_negotiation.cc:407] Connection header received
0621 07:52:04.727060 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:04.727079 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:04.727541 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:04.727881 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:04.728376 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:04.729421 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:04.730401 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:04.730952 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:04.731149 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:04.732017 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:04.732036 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:04.732049 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:04.732503 (+ 454us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:04.732564 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:04.732585 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:04.732693 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:04.732820 (+ 127us) server_negotiation.cc:299] Negotiation successful
0621 07:52:04.733097 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":141,"threads_started":1}
I20260621 07:52:04.736217 17528 raft_consensus.cc:348] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a064cce384494fd2a2766f6f9cd2bbe6" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42223 } }
I20260621 07:52:04.736512 17528 raft_consensus.cc:374] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:04.736595 17528 raft_consensus.cc:729] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a064cce384494fd2a2766f6f9cd2bbe6, State: Initialized, Role: FOLLOWER
I20260621 07:52:04.736948 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:52:04.737135 17528 consensus_queue.cc:260] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a064cce384494fd2a2766f6f9cd2bbe6" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42223 } }
I20260621 07:52:04.737295 17528 raft_consensus.cc:388] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:52:04.737391 17528 raft_consensus.cc:482] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:52:04.737498 17528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:04.738828 17528 raft_consensus.cc:504] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a064cce384494fd2a2766f6f9cd2bbe6" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42223 } }
I20260621 07:52:04.739185 17528 leader_election.cc:302] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a064cce384494fd2a2766f6f9cd2bbe6; no voters:
I20260621 07:52:04.739534 17528 leader_election.cc:288] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:52:04.739672 17533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:04.739926 17533 raft_consensus.cc:686] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [term 1 LEADER]: Becoming Leader. State: Replica: a064cce384494fd2a2766f6f9cd2bbe6, State: Running, Role: LEADER
I20260621 07:52:04.740239 17533 consensus_queue.cc:237] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a064cce384494fd2a2766f6f9cd2bbe6" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42223 } }
I20260621 07:52:04.740821 17528 sys_catalog.cc:564] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:52:04.741930 17535 sys_catalog.cc:455] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a064cce384494fd2a2766f6f9cd2bbe6. Latest consensus state: current_term: 1 leader_uuid: "a064cce384494fd2a2766f6f9cd2bbe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a064cce384494fd2a2766f6f9cd2bbe6" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42223 } } }
I20260621 07:52:04.742094 17535 sys_catalog.cc:458] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:04.742369 17534 sys_catalog.cc:455] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a064cce384494fd2a2766f6f9cd2bbe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a064cce384494fd2a2766f6f9cd2bbe6" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42223 } } }
I20260621 07:52:04.742486 17534 sys_catalog.cc:458] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6 [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:04.742758 17542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:52:04.745983 17542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:52:04.751844 17542 catalog_manager.cc:1347] Generated new cluster ID: f177d7ed4757457d8a824105cf3f888a
I20260621 07:52:04.751912 17542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:52:04.775347 17542 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:52:04.776645 17542 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:52:04.785368 17542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a064cce384494fd2a2766f6f9cd2bbe6: Generated new TSK 0
I20260621 07:52:04.786288 17542 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 07:52:04.814800 17558 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260621 07:52:04.814960 17558 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 07:52:04.815524 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:04.797727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42758 (local address 127.12.12.62:42223)
0621 07:52:04.797873 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:04.797878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:04.798050 (+ 172us) server_negotiation.cc:407] Connection header received
0621 07:52:04.798249 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:04.798252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:04.798312 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:04.798472 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0621 07:52:04.798801 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:04.799641 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:04.800527 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:04.800884 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:04.800984 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:04.801310 (+ 326us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 07:52:04.815098 (+ 13788us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 07:52:04.815384 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42758: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260621 07:52:04.815611 17531 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42758: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260621 07:52:04.816764 17557 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:04.797630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:42223 (local address 127.0.0.1:42758)
0621 07:52:04.797965 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:04.797977 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:04.798094 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:04.798551 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:04.798553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:04.798561 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0621 07:52:04.798683 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:04.798688 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:04.799764 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:04.799767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:04.800416 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:04.800423 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:04.801001 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:04.801004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:04.801080 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:04.801124 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 07:52:04.815362 (+ 14238us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 07:52:04.815605 (+ 243us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.12.62:42223: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":105,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.12.12.62:42223: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260621 07:52:04.817703 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 17461
2026-06-21T07:52:04Z chronyd exiting
I20260621 07:52:04.855657 12336 test_util.cc:175] -----------------------------------------------
I20260621 07:52:04.855777 12336 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028299065733-12336-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (331 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260621 07:52:04.856824 12336 mini_oidc.cc:152] Starting JWKS server
I20260621 07:52:04.857705 12336 webserver.cc:460] Webserver started at https://127.0.0.1:46151/ using document root <none> and password file <none>
I20260621 07:52:04.857838 12336 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 07:52:04.858104 12336 webserver.cc:460] Webserver started at http://127.0.0.1:35769/ using document root <none> and password file <none>
2026-06-21T07:52:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:52:04Z Disabled control of system clock
I20260621 07:52:04.867019 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:43581
--webserver_interface=127.12.12.62
--webserver_port=0
--builtin_ntp_servers=127.12.12.20:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:43581
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46151/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/testchainca.cert with env {}
W20260621 07:52:04.980370 17564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:04.980684 17564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:04.980727 17564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:04.984961 17564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:04.985033 17564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:04.985049 17564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:04.985066 17564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:04.985081 17564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 07:52:04.985098 17564 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 07:52:04.985118 17564 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46151/jwks.json
I20260621 07:52:04.990942 17564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:43581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:43581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46151/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:04.992172 17564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:04.993255 17564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:05.002519 17572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:05.002511 17570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:05.002511 17569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:05.002691 17564 server_base.cc:1034] running on GCE node
I20260621 07:52:05.003464 17564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:05.004559 17564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:05.005752 17564 hybrid_clock.cc:648] HybridClock initialized: now 1782028325005731 us; error 33 us; skew 500 ppm
I20260621 07:52:05.007982 17564 webserver.cc:460] Webserver started at http://127.12.12.62:42827/ using document root <none> and password file <none>
I20260621 07:52:05.008594 17564 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:05.008658 17564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:05.008875 17564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:05.011044 17564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "2979a7f1a5964fbf9482cab72c4f6e8b"
format_stamp: "Formatted at 2026-06-21 07:52:05 on dist-test-slave-lbpt"
server_key: "dbf9eafd83f865a7b5b5f36949d3086c"
server_key_iv: "dba0e4b1ed1f48d31f8b1b2df03b8f36"
server_key_version: "encryptionkey@0"
I20260621 07:52:05.011624 17564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "2979a7f1a5964fbf9482cab72c4f6e8b"
format_stamp: "Formatted at 2026-06-21 07:52:05 on dist-test-slave-lbpt"
server_key: "dbf9eafd83f865a7b5b5f36949d3086c"
server_key_iv: "dba0e4b1ed1f48d31f8b1b2df03b8f36"
server_key_version: "encryptionkey@0"
I20260621 07:52:05.015461 17564 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:52:05.018147 17578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:05.019546 17564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:52:05.019702 17564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "2979a7f1a5964fbf9482cab72c4f6e8b"
format_stamp: "Formatted at 2026-06-21 07:52:05 on dist-test-slave-lbpt"
server_key: "dbf9eafd83f865a7b5b5f36949d3086c"
server_key_iv: "dba0e4b1ed1f48d31f8b1b2df03b8f36"
server_key_version: "encryptionkey@0"
I20260621 07:52:05.019831 17564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:05.044624 17564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:05.045591 17564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:05.045809 17564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:05.053207 17564 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:43581
I20260621 07:52:05.053258 17630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:43581 every 8 connection(s)
I20260621 07:52:05.054235 17564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:52:05.057655 17631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:05.063359 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 17564
I20260621 07:52:05.063509 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:52:05.063798 12336 external_mini_cluster.cc:1428] Setting key f1d3c0d7a9d24f8d9f9fd94363f92246
I20260621 07:52:05.064296 17631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b: Bootstrap starting.
I20260621 07:52:05.066962 17631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:05.067852 17631 log.cc:824] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:05.069989 17631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b: No bootstrap required, opened a new log
I20260621 07:52:05.073468 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:05.065230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.12.12.62:43581)
0621 07:52:05.065896 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:05.065912 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:05.065950 (+ 38us) server_negotiation.cc:407] Connection header received
0621 07:52:05.066731 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:05.066752 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:05.067234 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:05.067533 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:05.067897 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.068914 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.070117 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.070696 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.070825 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:05.071681 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:05.071686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:05.071698 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:05.072102 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:05.072158 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:05.072177 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:05.072254 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:05.072425 (+ 171us) server_negotiation.cc:299] Negotiation successful
0621 07:52:05.072737 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":146,"threads_started":1}
I20260621 07:52:05.074138 17631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2979a7f1a5964fbf9482cab72c4f6e8b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43581 } }
I20260621 07:52:05.074347 17631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:05.074404 17631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2979a7f1a5964fbf9482cab72c4f6e8b, State: Initialized, Role: FOLLOWER
I20260621 07:52:05.074875 17631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2979a7f1a5964fbf9482cab72c4f6e8b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43581 } }
I20260621 07:52:05.074988 17631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:52:05.075029 17631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:52:05.075091 17631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:05.076054 17631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2979a7f1a5964fbf9482cab72c4f6e8b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43581 } }
I20260621 07:52:05.076306 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:52:05.076442 17631 leader_election.cc:302] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2979a7f1a5964fbf9482cab72c4f6e8b; no voters:
I20260621 07:52:05.076761 17631 leader_election.cc:288] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:52:05.076892 17636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:05.077129 17636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [term 1 LEADER]: Becoming Leader. State: Replica: 2979a7f1a5964fbf9482cab72c4f6e8b, State: Running, Role: LEADER
I20260621 07:52:05.077438 17636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2979a7f1a5964fbf9482cab72c4f6e8b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43581 } }
I20260621 07:52:05.078012 17631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:52:05.078827 17638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2979a7f1a5964fbf9482cab72c4f6e8b. Latest consensus state: current_term: 1 leader_uuid: "2979a7f1a5964fbf9482cab72c4f6e8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2979a7f1a5964fbf9482cab72c4f6e8b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43581 } } }
I20260621 07:52:05.078971 17638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:05.079217 17637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2979a7f1a5964fbf9482cab72c4f6e8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2979a7f1a5964fbf9482cab72c4f6e8b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43581 } } }
I20260621 07:52:05.079322 17645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:52:05.079341 17637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:05.082576 17645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:52:05.088374 17645 catalog_manager.cc:1347] Generated new cluster ID: e83513739f624df7b5f90c7f5c3a1385
I20260621 07:52:05.088475 17645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:52:05.104209 17645 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:52:05.105192 17645 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:52:05.115548 17645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2979a7f1a5964fbf9482cab72c4f6e8b: Generated new TSK 0
I20260621 07:52:05.116338 17645 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 07:52:05.145804 17661 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260621 07:52:05.145898 17661 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 07:52:05.146620 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:05.130580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39708 (local address 127.12.12.62:43581)
0621 07:52:05.130729 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:05.130734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:05.130981 (+ 247us) server_negotiation.cc:407] Connection header received
0621 07:52:05.131116 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:05.131118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:05.131168 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:05.131268 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0621 07:52:05.131525 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.132066 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.133320 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.133568 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.133657 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:05.134017 (+ 360us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 07:52:05.146211 (+ 12194us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46151/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 07:52:05.146495 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39708: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46151/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20260621 07:52:05.146701 17634 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39708: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46151/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260621 07:52:05.146859 17660 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:05.130549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43581 (local address 127.0.0.1:39708)
0621 07:52:05.130858 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:05.130874 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:05.130997 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:05.131289 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:05.131292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:05.131300 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0621 07:52:05.131414 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:05.131420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.132175 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.132179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:05.133178 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:05.133189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.133695 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.133699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:05.133794 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:05.133823 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 07:52:05.146455 (+ 12632us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46151/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 07:52:05.146670 (+ 215us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.12.62:43581: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46151/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":121,"threads_started":1}
I20260621 07:52:05.147552 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 17564
2026-06-21T07:52:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (329 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260621 07:52:05.186935 12336 mini_oidc.cc:152] Starting JWKS server
I20260621 07:52:05.187796 12336 webserver.cc:460] Webserver started at https://127.0.0.1:42653/ using document root <none> and password file <none>
I20260621 07:52:05.187879 12336 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 07:52:05.188093 12336 webserver.cc:460] Webserver started at http://127.0.0.1:36411/ using document root <none> and password file <none>
2026-06-21T07:52:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:52:05Z Disabled control of system clock
I20260621 07:52:05.197476 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:45877
--webserver_interface=127.12.12.62
--webserver_port=0
--builtin_ntp_servers=127.12.12.20:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:45877
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42653/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 07:52:05.307991 17667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:05.308284 17667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:05.308326 17667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:05.312369 17667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:05.312435 17667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:05.312451 17667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:05.312469 17667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:05.312484 17667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 07:52:05.312500 17667 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 07:52:05.312515 17667 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42653/jwks.json
I20260621 07:52:05.317418 17667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:45877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:45877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42653/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:05.318571 17667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:05.319986 17667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:05.328842 17673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:05.328848 17672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:05.328853 17675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:05.329161 17667 server_base.cc:1034] running on GCE node
I20260621 07:52:05.329754 17667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:05.330974 17667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:05.332166 17667 hybrid_clock.cc:648] HybridClock initialized: now 1782028325332145 us; error 36 us; skew 500 ppm
I20260621 07:52:05.334062 17667 webserver.cc:460] Webserver started at http://127.12.12.62:38277/ using document root <none> and password file <none>
I20260621 07:52:05.334712 17667 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:05.334806 17667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:05.335041 17667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:05.337055 17667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "52f6fecf76264bbb8e41ab554fe53626"
format_stamp: "Formatted at 2026-06-21 07:52:05 on dist-test-slave-lbpt"
server_key: "2c82892ea6ec4a36d69336c8af03e625"
server_key_iv: "dd4fdc65878525eed29b384b110a989b"
server_key_version: "encryptionkey@0"
I20260621 07:52:05.337620 17667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "52f6fecf76264bbb8e41ab554fe53626"
format_stamp: "Formatted at 2026-06-21 07:52:05 on dist-test-slave-lbpt"
server_key: "2c82892ea6ec4a36d69336c8af03e625"
server_key_iv: "dd4fdc65878525eed29b384b110a989b"
server_key_version: "encryptionkey@0"
I20260621 07:52:05.341331 17667 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260621 07:52:05.343743 17681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:05.344952 17667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:52:05.345090 17667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "52f6fecf76264bbb8e41ab554fe53626"
format_stamp: "Formatted at 2026-06-21 07:52:05 on dist-test-slave-lbpt"
server_key: "2c82892ea6ec4a36d69336c8af03e625"
server_key_iv: "dd4fdc65878525eed29b384b110a989b"
server_key_version: "encryptionkey@0"
I20260621 07:52:05.345202 17667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:05.363736 17667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:05.364471 17667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:05.364673 17667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:05.371803 17667 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:45877
I20260621 07:52:05.371820 17733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:45877 every 8 connection(s)
I20260621 07:52:05.372781 17667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:52:05.373344 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 17667
I20260621 07:52:05.373473 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:52:05.373759 12336 external_mini_cluster.cc:1428] Setting key 06a8a3048cc6601cfcb91ce28529cc0f
I20260621 07:52:05.376049 17734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:05.382592 17734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626: Bootstrap starting.
I20260621 07:52:05.383394 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:05.375243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.12.12.62:45877)
0621 07:52:05.375891 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:05.375907 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:05.375942 (+ 35us) server_negotiation.cc:407] Connection header received
0621 07:52:05.376636 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:05.376654 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:05.377063 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:05.377350 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:05.377683 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.378728 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.379958 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.380451 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.380582 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:05.381518 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:05.381523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:05.381536 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:05.381958 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:05.382015 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:05.382034 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:05.382111 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:05.382275 (+ 164us) server_negotiation.cc:299] Negotiation successful
0621 07:52:05.382605 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":142,"threads_started":1}
I20260621 07:52:05.385100 17734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:05.385998 17734 log.cc:824] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:05.386493 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:52:05.388410 17734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626: No bootstrap required, opened a new log
I20260621 07:52:05.392175 17734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f6fecf76264bbb8e41ab554fe53626" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 45877 } }
I20260621 07:52:05.392432 17734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:05.392515 17734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f6fecf76264bbb8e41ab554fe53626, State: Initialized, Role: FOLLOWER
I20260621 07:52:05.392940 17734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f6fecf76264bbb8e41ab554fe53626" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 45877 } }
I20260621 07:52:05.393088 17734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:52:05.393174 17734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:52:05.393270 17734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:05.394280 17734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f6fecf76264bbb8e41ab554fe53626" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 45877 } }
I20260621 07:52:05.394660 17734 leader_election.cc:302] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52f6fecf76264bbb8e41ab554fe53626; no voters:
I20260621 07:52:05.394976 17734 leader_election.cc:288] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:52:05.395150 17739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:05.395393 17739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [term 1 LEADER]: Becoming Leader. State: Replica: 52f6fecf76264bbb8e41ab554fe53626, State: Running, Role: LEADER
I20260621 07:52:05.395722 17739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f6fecf76264bbb8e41ab554fe53626" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 45877 } }
I20260621 07:52:05.396230 17734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:52:05.397287 17741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52f6fecf76264bbb8e41ab554fe53626. Latest consensus state: current_term: 1 leader_uuid: "52f6fecf76264bbb8e41ab554fe53626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f6fecf76264bbb8e41ab554fe53626" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 45877 } } }
I20260621 07:52:05.397416 17741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:05.397714 17740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52f6fecf76264bbb8e41ab554fe53626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f6fecf76264bbb8e41ab554fe53626" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 45877 } } }
I20260621 07:52:05.397802 17740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626 [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:05.398049 17748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:52:05.400753 17748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:52:05.406513 17748 catalog_manager.cc:1347] Generated new cluster ID: eaeb2b021ee74d9da649df036babbf7c
I20260621 07:52:05.406589 17748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:52:05.428865 17748 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:52:05.429903 17748 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:52:05.447939 17748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52f6fecf76264bbb8e41ab554fe53626: Generated new TSK 0
I20260621 07:52:05.448727 17748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:52:05.468885 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:05.464002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.12.12.62:45877)
0621 07:52:05.464147 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:05.464150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:05.464265 (+ 115us) server_negotiation.cc:407] Connection header received
0621 07:52:05.464447 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:05.464449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:05.464498 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:05.464608 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0621 07:52:05.464861 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.465371 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.466646 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.466864 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.466943 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:05.467273 (+ 330us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 07:52:05.468477 (+ 1204us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42653/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 07:52:05.468726 (+ 249us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42653/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260621 07:52:05.468971 17763 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:05.463841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:45877 (local address 127.0.0.1:50166)
0621 07:52:05.464169 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:05.464184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:05.464302 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:05.464624 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:05.464627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:05.464636 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0621 07:52:05.464755 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:05.464761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.465506 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.465510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:05.466534 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:05.466544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.466980 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.466984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:05.467075 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:05.467104 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 07:52:05.468643 (+ 1539us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42653/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 07:52:05.468795 (+ 152us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.12.62:45877: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42653/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":105,"threads_started":1}
W20260621 07:52:05.468972 17737 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42653/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260621 07:52:05.469714 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 17667
E20260621 07:52:05.479779 17764 webserver.cc:533] Webserver: sslize failed:
E20260621 07:52:05.479861 17764 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-21T07:52:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (322 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-21T07:52:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:52:05Z Disabled control of system clock
I20260621 07:52:05.518121 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:44467
--webserver_interface=127.12.12.62
--webserver_port=0
--builtin_ntp_servers=127.12.12.20:46303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:44467
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/testchainca.cert with env {}
W20260621 07:52:05.630559 17768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:05.630842 17768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:05.630882 17768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:05.635063 17768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:05.635133 17768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:05.635149 17768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:05.635166 17768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:05.635181 17768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 07:52:05.635198 17768 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 07:52:05.635211 17768 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260621 07:52:05.640691 17768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:46303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:44467
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:44467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:05.641763 17768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:05.642844 17768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:05.651908 17773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:05.651907 17776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:05.651908 17774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:05.652235 17768 server_base.cc:1034] running on GCE node
I20260621 07:52:05.652818 17768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:05.653972 17768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:05.655133 17768 hybrid_clock.cc:648] HybridClock initialized: now 1782028325655119 us; error 27 us; skew 500 ppm
I20260621 07:52:05.657117 17768 webserver.cc:460] Webserver started at http://127.12.12.62:37845/ using document root <none> and password file <none>
I20260621 07:52:05.657727 17768 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:05.657789 17768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:05.658007 17768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:05.660159 17768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "959f7b819a9b4e8a9e592ee4df04252b"
format_stamp: "Formatted at 2026-06-21 07:52:05 on dist-test-slave-lbpt"
server_key: "4fa4f6bc91536cbaa46c8bfe9a0774ac"
server_key_iv: "0b8e77722e7baa83d08679d5c067c5c3"
server_key_version: "encryptionkey@0"
I20260621 07:52:05.660748 17768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "959f7b819a9b4e8a9e592ee4df04252b"
format_stamp: "Formatted at 2026-06-21 07:52:05 on dist-test-slave-lbpt"
server_key: "4fa4f6bc91536cbaa46c8bfe9a0774ac"
server_key_iv: "0b8e77722e7baa83d08679d5c067c5c3"
server_key_version: "encryptionkey@0"
I20260621 07:52:05.664757 17768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260621 07:52:05.667475 17782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:05.669010 17768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:05.669198 17768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "959f7b819a9b4e8a9e592ee4df04252b"
format_stamp: "Formatted at 2026-06-21 07:52:05 on dist-test-slave-lbpt"
server_key: "4fa4f6bc91536cbaa46c8bfe9a0774ac"
server_key_iv: "0b8e77722e7baa83d08679d5c067c5c3"
server_key_version: "encryptionkey@0"
I20260621 07:52:05.669348 17768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:05.680732 17768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:05.681494 17768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:05.681720 17768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:05.689182 17768 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:44467
I20260621 07:52:05.689208 17834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:44467 every 8 connection(s)
I20260621 07:52:05.690158 17768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:52:05.693076 17835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:05.694105 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 17768
I20260621 07:52:05.694259 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:52:05.694630 12336 external_mini_cluster.cc:1428] Setting key 658edc96bb7946908e46a1d4b02d5e86
I20260621 07:52:05.699558 17835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b: Bootstrap starting.
I20260621 07:52:05.702534 17835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:05.703367 17835 log.cc:824] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:05.704653 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:05.696093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.12.12.62:44467)
0621 07:52:05.696817 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:05.696838 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:05.696875 (+ 37us) server_negotiation.cc:407] Connection header received
0621 07:52:05.697524 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:05.697542 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:05.697962 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:05.698246 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:05.698694 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.699662 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.700903 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.701405 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.701551 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:05.702580 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:05.702585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:05.702598 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:05.703062 (+ 464us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:05.703123 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:05.703145 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:05.703231 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:05.703406 (+ 175us) server_negotiation.cc:299] Negotiation successful
0621 07:52:05.703734 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":163,"threads_started":1}
I20260621 07:52:05.705470 17835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b: No bootstrap required, opened a new log
I20260621 07:52:05.707561 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:52:05.709564 17835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "959f7b819a9b4e8a9e592ee4df04252b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44467 } }
I20260621 07:52:05.709820 17835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:05.709908 17835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 959f7b819a9b4e8a9e592ee4df04252b, State: Initialized, Role: FOLLOWER
I20260621 07:52:05.710369 17835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "959f7b819a9b4e8a9e592ee4df04252b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44467 } }
I20260621 07:52:05.710575 17835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:52:05.710662 17835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:52:05.710762 17835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:05.711812 17835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "959f7b819a9b4e8a9e592ee4df04252b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44467 } }
I20260621 07:52:05.712172 17835 leader_election.cc:302] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 959f7b819a9b4e8a9e592ee4df04252b; no voters:
I20260621 07:52:05.712531 17835 leader_election.cc:288] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:52:05.712684 17840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:05.712900 17840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [term 1 LEADER]: Becoming Leader. State: Replica: 959f7b819a9b4e8a9e592ee4df04252b, State: Running, Role: LEADER
I20260621 07:52:05.713246 17840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "959f7b819a9b4e8a9e592ee4df04252b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44467 } }
I20260621 07:52:05.713886 17835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:52:05.714985 17841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "959f7b819a9b4e8a9e592ee4df04252b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "959f7b819a9b4e8a9e592ee4df04252b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44467 } } }
I20260621 07:52:05.715152 17841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:05.715466 17842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 959f7b819a9b4e8a9e592ee4df04252b. Latest consensus state: current_term: 1 leader_uuid: "959f7b819a9b4e8a9e592ee4df04252b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "959f7b819a9b4e8a9e592ee4df04252b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44467 } } }
I20260621 07:52:05.715561 17842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b [sys.catalog]: This master's current role is: LEADER
W20260621 07:52:05.718890 17856 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:52:05.718978 17856 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:52:05.719049 17849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:52:05.719810 17849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:52:05.725700 17849 catalog_manager.cc:1347] Generated new cluster ID: 8bfca1a88cae454f9eb8145c76fe9602
I20260621 07:52:05.725780 17849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:52:05.740912 17849 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:52:05.741856 17849 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:52:05.749401 17849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 959f7b819a9b4e8a9e592ee4df04252b: Generated new TSK 0
I20260621 07:52:05.750136 17849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:52:05.834790 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:05.765779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40448 (local address 127.12.12.62:44467)
0621 07:52:05.765917 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:05.765924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:05.766113 (+ 189us) server_negotiation.cc:407] Connection header received
0621 07:52:05.766358 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:05.766361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:05.766496 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:05.766622 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0621 07:52:05.766875 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.767466 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.768715 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.768932 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.769017 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:05.769463 (+ 446us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 07:52:05.834280 (+ 64817us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 07:52:05.834666 (+ 386us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40448: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20260621 07:52:05.834890 17838 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40448: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260621 07:52:05.835038 17864 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:05.765690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:44467 (local address 127.0.0.1:40448)
0621 07:52:05.766017 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:05.766032 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:05.766152 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:05.766630 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:05.766633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:05.766642 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0621 07:52:05.766757 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:05.766763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.767582 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.767585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:05.768581 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:05.768590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:05.769060 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:05.769065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:05.769163 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:05.769190 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 07:52:05.834571 (+ 65381us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 07:52:05.834820 (+ 249us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.12.62:44467: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":106,"threads_started":1}
I20260621 07:52:05.835721 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 17768
2026-06-21T07:52:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (345 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (125 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (128 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:52:06 dist-test-slave-lbpt krb5kdc[17876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:52:06 dist-test-slave-lbpt krb5kdc[17876](info): set up 2 sockets
Jun 21 07:52:06 dist-test-slave-lbpt krb5kdc[17876](info): commencing operation
krb5kdc: starting...
W20260621 07:52:07.142897 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:52:07 dist-test-slave-lbpt krb5kdc[17876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:52:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:52:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:52:07.269994 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:40527
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:40527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:07.385169 17889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:07.385532 17889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:07.385617 17889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:07.389703 17889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:07.389765 17889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:07.389784 17889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:07.389801 17889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:07.389848 17889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:52:07.395212 17889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:40527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:40527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:07.396304 17889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:07.397334 17889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:07.405666 17895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:07.405666 17894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:07.405668 17897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:07.405829 17889 server_base.cc:1034] running on GCE node
I20260621 07:52:07.406564 17889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:07.407720 17889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:07.408932 17889 hybrid_clock.cc:648] HybridClock initialized: now 1782028327408902 us; error 43 us; skew 500 ppm
Jun 21 07:52:07 dist-test-slave-lbpt krb5kdc[17876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028327, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:07.414978 17889 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:52:07.416009 17889 webserver.cc:460] Webserver started at http://127.12.12.62:39599/ using document root <none> and password file <none>
I20260621 07:52:07.416575 17889 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:07.416664 17889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:07.416904 17889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:07.419031 17889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "155642c183944e1bbf14e98812b4819b"
format_stamp: "Formatted at 2026-06-21 07:52:07 on dist-test-slave-lbpt"
server_key: "86d5260855a6226818666a5e3ab696d0"
server_key_iv: "efcbe0d7b3694ed22f6935048bd0e55a"
server_key_version: "encryptionkey@0"
I20260621 07:52:07.419585 17889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "155642c183944e1bbf14e98812b4819b"
format_stamp: "Formatted at 2026-06-21 07:52:07 on dist-test-slave-lbpt"
server_key: "86d5260855a6226818666a5e3ab696d0"
server_key_iv: "efcbe0d7b3694ed22f6935048bd0e55a"
server_key_version: "encryptionkey@0"
I20260621 07:52:07.423615 17889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:52:07.426121 17904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:07.427495 17889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:07.427644 17889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "155642c183944e1bbf14e98812b4819b"
format_stamp: "Formatted at 2026-06-21 07:52:07 on dist-test-slave-lbpt"
server_key: "86d5260855a6226818666a5e3ab696d0"
server_key_iv: "efcbe0d7b3694ed22f6935048bd0e55a"
server_key_version: "encryptionkey@0"
I20260621 07:52:07.427762 17889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:07.448707 17889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:07.452283 17889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:07.452515 17889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:07.459574 17889 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:40527
I20260621 07:52:07.459597 17956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:40527 every 8 connection(s)
I20260621 07:52:07.460554 17889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:52:07.463385 17957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:07.466843 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 17889
I20260621 07:52:07.466964 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:52:07.467231 12336 external_mini_cluster.cc:1428] Setting key acff0c227f8c0842324c4074109cbcfa
I20260621 07:52:07.469324 17957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b: Bootstrap starting.
I20260621 07:52:07.471527 17957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:07.472249 17957 log.cc:824] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:07.474028 17957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b: No bootstrap required, opened a new log
Jun 21 07:52:07 dist-test-slave-lbpt krb5kdc[17876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:07.477803 17957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155642c183944e1bbf14e98812b4819b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 40527 } }
I20260621 07:52:07.478009 17957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:07.478088 17957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155642c183944e1bbf14e98812b4819b, State: Initialized, Role: FOLLOWER
I20260621 07:52:07.478562 17957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155642c183944e1bbf14e98812b4819b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 40527 } }
I20260621 07:52:07.478683 17957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:52:07.478767 17957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:52:07.478881 17957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:07.479700 17957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155642c183944e1bbf14e98812b4819b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 40527 } }
I20260621 07:52:07.480039 17957 leader_election.cc:302] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 155642c183944e1bbf14e98812b4819b; no voters:
I20260621 07:52:07.480367 17957 leader_election.cc:288] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:52:07.480443 17962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:07.480641 17962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [term 1 LEADER]: Becoming Leader. State: Replica: 155642c183944e1bbf14e98812b4819b, State: Running, Role: LEADER
I20260621 07:52:07.480967 17962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155642c183944e1bbf14e98812b4819b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 40527 } }
I20260621 07:52:07.481637 17957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:52:07.481875 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:07.468698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.12.12.62:40527)
0621 07:52:07.469142 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:07.469157 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:07.469193 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:52:07.469885 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:07.469903 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:07.470249 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:07.470642 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:07.471457 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:07.472554 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:07.473406 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:07.473877 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:07.474061 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:07.476794 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:07.476800 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:07.476813 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:07.476839 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:07.478948 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:07.479379 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:07.479384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:07.479390 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:07.479513 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:07.479817 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:07.479820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:07.479822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:07.480203 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:07.480417 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:07.480711 (+ 294us) server_negotiation.cc:299] Negotiation successful
0621 07:52:07.481105 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":130,"threads_started":1}
I20260621 07:52:07.482188 17962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 155642c183944e1bbf14e98812b4819b. Latest consensus state: current_term: 1 leader_uuid: "155642c183944e1bbf14e98812b4819b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155642c183944e1bbf14e98812b4819b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 40527 } } }
I20260621 07:52:07.482300 17962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:07.482620 17963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "155642c183944e1bbf14e98812b4819b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155642c183944e1bbf14e98812b4819b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 40527 } } }
I20260621 07:52:07.482723 17963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:07.482874 17968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:52:07.486662 17968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:52:07.492321 17968 catalog_manager.cc:1347] Generated new cluster ID: e03c2a620b0848a280853bd163e6f00e
I20260621 07:52:07.492396 17968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:52:07.524974 17968 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:52:07.526266 17968 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:52:07.530130 17968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 155642c183944e1bbf14e98812b4819b: Generated new TSK 0
I20260621 07:52:07.530929 17968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:52:07.566291 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:40527
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:07.699523 17983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:07.699796 17983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:07.699841 17983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:07.704123 17983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:07.704192 17983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:07.704254 17983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:52:07.709632 17983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:40527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:07.710804 17983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:07.711850 17983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:07.721256 17991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:07.721279 17988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:07.721282 17989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:07.721616 17983 server_base.cc:1034] running on GCE node
I20260621 07:52:07.722018 17983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:07.722690 17983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:07.723894 17983 hybrid_clock.cc:648] HybridClock initialized: now 1782028327723857 us; error 63 us; skew 500 ppm
Jun 21 07:52:07 dist-test-slave-lbpt krb5kdc[17876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028327, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:07.726706 17983 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:52:07.727509 17983 webserver.cc:460] Webserver started at http://127.12.12.1:43389/ using document root <none> and password file <none>
I20260621 07:52:07.728024 17983 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:07.728080 17983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:07.728242 17983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:07.730208 17983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "e3706c96def24db6a574e128122fd974"
format_stamp: "Formatted at 2026-06-21 07:52:07 on dist-test-slave-lbpt"
server_key: "b4a18b5ea4ebf98634086a2590e26c76"
server_key_iv: "2f6873636ac543dc16a74b023286c826"
server_key_version: "encryptionkey@0"
I20260621 07:52:07.730798 17983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "e3706c96def24db6a574e128122fd974"
format_stamp: "Formatted at 2026-06-21 07:52:07 on dist-test-slave-lbpt"
server_key: "b4a18b5ea4ebf98634086a2590e26c76"
server_key_iv: "2f6873636ac543dc16a74b023286c826"
server_key_version: "encryptionkey@0"
I20260621 07:52:07.734647 17983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 07:52:07.737069 17998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:07.738301 17983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:52:07.738462 17983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "e3706c96def24db6a574e128122fd974"
format_stamp: "Formatted at 2026-06-21 07:52:07 on dist-test-slave-lbpt"
server_key: "b4a18b5ea4ebf98634086a2590e26c76"
server_key_iv: "2f6873636ac543dc16a74b023286c826"
server_key_version: "encryptionkey@0"
I20260621 07:52:07.738586 17983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:07.752642 17983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:07.755873 17983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:07.756079 17983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:07.756696 17983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:07.757521 17983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:07.757591 17983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:07.757661 17983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:07.757714 17983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:07.766665 17983 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:33321
I20260621 07:52:07.766741 18111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:33321 every 8 connection(s)
I20260621 07:52:07.767640 17983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:52:07.773953 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 17983
I20260621 07:52:07.774127 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:52:07.774399 12336 external_mini_cluster.cc:1428] Setting key 9e8ba1748ec1d3ac1e22400fbac8465c
Jun 21 07:52:07 dist-test-slave-lbpt krb5kdc[17876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028327, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:07.781888 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:07.769616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:47697 (local address 127.12.12.62:40527)
0621 07:52:07.769754 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:07.769758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:07.770726 (+ 968us) server_negotiation.cc:407] Connection header received
0621 07:52:07.772070 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:07.772073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:07.772128 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:07.772210 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:07.773356 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:07.773890 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:07.775346 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:07.775653 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:07.775770 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:07.778217 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:07.778228 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:07.778231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:07.778254 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:07.779708 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:07.780211 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:07.780216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:07.780218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:07.780298 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:07.780649 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:07.780652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:07.780654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:07.780832 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:07.780934 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:07.781540 (+ 606us) server_negotiation.cc:299] Negotiation successful
0621 07:52:07.781686 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:52:07.782743 18114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:07.770023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:40527 (local address 127.12.12.1:47697)
0621 07:52:07.770577 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:07.770621 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:07.771859 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:07.772373 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:07.772380 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:07.772740 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:07.773208 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:07.773222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:07.774025 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:07.774036 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:07.775230 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:07.775240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:07.775806 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:07.775810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:07.775985 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:07.776583 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:07.776599 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:07.778055 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:07.779824 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:07.779830 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:07.779845 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:07.780108 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:07.780390 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:07.780394 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:07.780397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:07.780546 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:07.780944 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:07.780953 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:07.781249 (+ 296us) client_negotiation.cc:769] Sending connection context
0621 07:52:07.781461 (+ 212us) client_negotiation.cc:240] Negotiation successful
0621 07:52:07.781724 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
I20260621 07:52:07.784049 18112 heartbeater.cc:347] Connected to a master server at 127.12.12.62:40527
I20260621 07:52:07.784297 18112 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:07.784821 18112 heartbeater.cc:511] Master 127.12.12.62:40527 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
I20260621 07:52:07.786523 17921 ts_manager.cc:194] Registered new tserver with Master: e3706c96def24db6a574e128122fd974 (127.12.12.1:33321)
I20260621 07:52:07.787642 17921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:47697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:52:07.801934 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:40527
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:07.915249 18117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:07.915586 18117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:07.915663 18117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:07.920022 18117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:07.920111 18117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:07.920209 18117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:52:07.925807 18117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:40527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:07.927052 18117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:07.928206 18117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:07.938628 18122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:07.938633 18123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:07.938629 18125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:07.938913 18117 server_base.cc:1034] running on GCE node
I20260621 07:52:07.939433 18117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:07.940071 18117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:07.941239 18117 hybrid_clock.cc:648] HybridClock initialized: now 1782028327941203 us; error 50 us; skew 500 ppm
Jun 21 07:52:07 dist-test-slave-lbpt krb5kdc[17876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028327, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:07.944244 18117 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:52:07.945181 18117 webserver.cc:460] Webserver started at http://127.12.12.2:33703/ using document root <none> and password file <none>
I20260621 07:52:07.945747 18117 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:07.945837 18117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:07.946074 18117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:07.948248 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "475be89f5b904563bfbb2d4fc00e5deb"
format_stamp: "Formatted at 2026-06-21 07:52:07 on dist-test-slave-lbpt"
server_key: "9bc28b7146c6b87753e11969543939f2"
server_key_iv: "13f16565ca9957f62442dcc9db77ecb0"
server_key_version: "encryptionkey@0"
I20260621 07:52:07.948833 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "475be89f5b904563bfbb2d4fc00e5deb"
format_stamp: "Formatted at 2026-06-21 07:52:07 on dist-test-slave-lbpt"
server_key: "9bc28b7146c6b87753e11969543939f2"
server_key_iv: "13f16565ca9957f62442dcc9db77ecb0"
server_key_version: "encryptionkey@0"
I20260621 07:52:07.952868 18117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:52:07.955541 18132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:07.956980 18117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:07.957115 18117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "475be89f5b904563bfbb2d4fc00e5deb"
format_stamp: "Formatted at 2026-06-21 07:52:07 on dist-test-slave-lbpt"
server_key: "9bc28b7146c6b87753e11969543939f2"
server_key_iv: "13f16565ca9957f62442dcc9db77ecb0"
server_key_version: "encryptionkey@0"
I20260621 07:52:07.957231 18117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:07.972486 18117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:07.975996 18117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:07.976209 18117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:07.976820 18117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:07.977663 18117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:07.977738 18117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:07.977809 18117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:07.977854 18117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:07.987443 18117 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:42953
I20260621 07:52:07.987502 18245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:42953 every 8 connection(s)
I20260621 07:52:07.988382 18117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:52:07.988701 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 18117
I20260621 07:52:07.988816 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:52:07.989171 12336 external_mini_cluster.cc:1428] Setting key b1e8a15b6cec925d79cb33437e1313d8
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
Jun 21 07:52:07 dist-test-slave-lbpt krb5kdc[17876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028327, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:08.008960 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:07.990285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:57331 (local address 127.12.12.62:40527)
0621 07:52:07.990449 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:07.990455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:07.991585 (+ 1130us) server_negotiation.cc:407] Connection header received
0621 07:52:07.993579 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:07.993582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:07.993668 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:07.993763 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:07.995674 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:07.996681 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:07.998377 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:07.998837 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:07.998943 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:08.004781 (+ 5838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:08.004788 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:08.004794 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:08.004835 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:08.006480 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:08.007109 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:08.007116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:08.007120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:08.007219 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:08.007612 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:08.007618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:08.007623 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:08.007909 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:08.008049 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:08.008577 (+ 528us) server_negotiation.cc:299] Negotiation successful
0621 07:52:08.008739 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 07:52:08.010186 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:07.990842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:40527 (local address 127.12.12.2:57331)
0621 07:52:07.991390 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:07.991454 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:07.993289 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:07.994020 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:07.994030 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:07.994649 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:07.995461 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:07.995485 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:07.996867 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:07.996870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:07.998223 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:07.998235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:07.999013 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:07.999017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:07.999200 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:07.999828 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:07.999850 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:08.004562 (+ 4712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:08.006618 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:08.006628 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:08.006651 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:08.006968 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:08.007312 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:08.007314 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:08.007316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:08.007493 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:08.008038 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:08.008044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:08.008302 (+ 258us) client_negotiation.cc:769] Sending connection context
0621 07:52:08.008477 (+ 175us) client_negotiation.cc:240] Negotiation successful
0621 07:52:08.008928 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":120,"threads_started":1}
I20260621 07:52:08.011595 18246 heartbeater.cc:347] Connected to a master server at 127.12.12.62:40527
I20260621 07:52:08.011860 18246 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:08.012392 18246 heartbeater.cc:511] Master 127.12.12.62:40527 requested a full tablet report, sending...
I20260621 07:52:08.013540 17921 ts_manager.cc:194] Registered new tserver with Master: 475be89f5b904563bfbb2d4fc00e5deb (127.12.12.2:42953)
I20260621 07:52:08.014206 17921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:57331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:52:08.019665 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:40527
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:08.131547 18251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:08.131825 18251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:08.131870 18251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:08.136018 18251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:08.136082 18251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:08.136144 18251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:52:08.141604 18251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:40527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:08.142817 18251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:08.143934 18251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:08.154517 18256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:08.154517 18259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:08.154626 18251 server_base.cc:1034] running on GCE node
W20260621 07:52:08.154517 18257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:08.155350 18251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:08.156085 18251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:08.157300 18251 hybrid_clock.cc:648] HybridClock initialized: now 1782028328157274 us; error 37 us; skew 500 ppm
Jun 21 07:52:08 dist-test-slave-lbpt krb5kdc[17876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028328, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:08.160470 18251 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:52:08.161453 18251 webserver.cc:460] Webserver started at http://127.12.12.3:35249/ using document root <none> and password file <none>
I20260621 07:52:08.162006 18251 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:08.162068 18251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:08.162240 18251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:08.164377 18251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "c2989b59cd3d4e4490092a2aeabefff8"
format_stamp: "Formatted at 2026-06-21 07:52:08 on dist-test-slave-lbpt"
server_key: "f0dcc7cf4d2b07cf134ecc4e7bb1dbbd"
server_key_iv: "7bab7458507c3b38cd63783e6239421d"
server_key_version: "encryptionkey@0"
I20260621 07:52:08.164891 18251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "c2989b59cd3d4e4490092a2aeabefff8"
format_stamp: "Formatted at 2026-06-21 07:52:08 on dist-test-slave-lbpt"
server_key: "f0dcc7cf4d2b07cf134ecc4e7bb1dbbd"
server_key_iv: "7bab7458507c3b38cd63783e6239421d"
server_key_version: "encryptionkey@0"
I20260621 07:52:08.168753 18251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:52:08.171408 18266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:08.172716 18251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:08.172818 18251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "c2989b59cd3d4e4490092a2aeabefff8"
format_stamp: "Formatted at 2026-06-21 07:52:08 on dist-test-slave-lbpt"
server_key: "f0dcc7cf4d2b07cf134ecc4e7bb1dbbd"
server_key_iv: "7bab7458507c3b38cd63783e6239421d"
server_key_version: "encryptionkey@0"
I20260621 07:52:08.172910 18251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:08.183761 18251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:08.186947 18251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:08.187104 18251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:08.187688 18251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:08.188477 18251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:08.188522 18251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:08.188560 18251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:08.188575 18251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:08.197731 18251 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:43199
I20260621 07:52:08.197753 18379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:43199 every 8 connection(s)
I20260621 07:52:08.198761 18251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:52:08.206346 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 18251
I20260621 07:52:08.206497 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:52:08.206856 12336 external_mini_cluster.cc:1428] Setting key daf6ede567012de53964e664519bf197
Jun 21 07:52:08 dist-test-slave-lbpt krb5kdc[17876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028328, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:08.214046 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:08.200679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:34147 (local address 127.12.12.62:40527)
0621 07:52:08.200837 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:08.200842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:08.201725 (+ 883us) server_negotiation.cc:407] Connection header received
0621 07:52:08.203352 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:08.203355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:08.203416 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:08.203522 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:08.204833 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:08.205491 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:08.206833 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:08.207134 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:08.207249 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:08.210284 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:08.210287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:08.210290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:08.210350 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:08.211933 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:08.212492 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:08.212495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:08.212497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:08.212585 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:08.212934 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:08.212937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:08.212938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:08.213132 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:08.213231 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:08.213795 (+ 564us) server_negotiation.cc:299] Negotiation successful
0621 07:52:08.213903 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 07:52:08.215117 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:08.201066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:40527 (local address 127.12.12.3:34147)
0621 07:52:08.201561 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:08.201606 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:08.203117 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:08.203706 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:08.203713 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:08.204176 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:08.204676 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:08.204691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:08.205626 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:08.205629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:08.206704 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:08.206714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:08.207295 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:08.207299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:08.207513 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:08.208320 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:08.208350 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:08.210114 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:08.212064 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:08.212070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:08.212085 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:08.212389 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:08.212671 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:08.212673 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:08.212675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:08.212853 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:08.213255 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:08.213261 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:08.213573 (+ 312us) client_negotiation.cc:769] Sending connection context
0621 07:52:08.213751 (+ 178us) client_negotiation.cc:240] Negotiation successful
0621 07:52:08.214047 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":127,"threads_started":1}
I20260621 07:52:08.216404 18380 heartbeater.cc:347] Connected to a master server at 127.12.12.62:40527
I20260621 07:52:08.216717 18380 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:08.217288 18380 heartbeater.cc:511] Master 127.12.12.62:40527 requested a full tablet report, sending...
I20260621 07:52:08.218585 17921 ts_manager.cc:194] Registered new tserver with Master: c2989b59cd3d4e4490092a2aeabefff8 (127.12.12.3:43199)
I20260621 07:52:08.219193 17921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:34147
I20260621 07:52:08.221212 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/test-admin.keytab.
Jun 21 07:52:08 dist-test-slave-lbpt krb5kdc[17876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:08.239979 12336 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260621 07:52:08.243144 18387 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:08.241504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.12.12.1:33321)
0621 07:52:08.241881 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:08.241897 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:08.241919 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:52:08.241969 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:08.241974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:08.242164 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:08.242380 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:08.242952 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34534: BlockingRecv error: recv got EOF from 127.0.0.1:34534 (error 108)
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":128,"threads_started":1}
W20260621 07:52:08.243270 12336 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/krb5cc' of type FILE
Jun 21 07:52:08 dist-test-slave-lbpt krb5kdc[17876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:08.244367 12336 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 07:52:08 dist-test-slave-lbpt krb5kdc[17876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:08.245340 12336 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 07:52:08 dist-test-slave-lbpt krb5kdc[17876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.1@KRBTEST.COM
I20260621 07:52:08.255633 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:08.246260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.12.12.1:33321)
0621 07:52:08.246518 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:08.246524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:08.246613 (+ 89us) server_negotiation.cc:407] Connection header received
0621 07:52:08.246776 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:08.246780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:08.246828 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:08.246914 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:08.247467 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:08.248279 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:08.249153 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:08.249565 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:08.249684 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:08.252225 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:08.252235 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:08.252249 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:08.252274 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:08.253747 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:08.254207 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:08.254221 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:08.254223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:08.254302 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:08.254623 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:08.254627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:08.254629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:08.254891 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:08.255131 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:08.255319 (+ 188us) server_negotiation.cc:299] Negotiation successful
0621 07:52:08.255481 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260621 07:52:08.259884 18387 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:08.258767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.12.12.1:33321)
0621 07:52:08.258905 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:08.258909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:08.259035 (+ 126us) server_negotiation.cc:407] Connection header received
0621 07:52:08.259181 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:08.259184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:08.259225 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:08.259317 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:08.259753 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34546: BlockingRecv error: recv got EOF from 127.0.0.1:34546 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260621 07:52:08.260016 12336 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/krb5cc' of type FILE
Jun 21 07:52:08 dist-test-slave-lbpt krb5kdc[17876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:08.261143 12336 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 07:52:08 dist-test-slave-lbpt krb5kdc[17876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:08.262260 12336 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 07:52:08 dist-test-slave-lbpt krb5kdc[17876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.1@KRBTEST.COM
I20260621 07:52:08.271073 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:08.263169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34556 (local address 127.12.12.1:33321)
0621 07:52:08.263293 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:08.263302 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:08.263314 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:52:08.263417 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:08.263419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:08.263455 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:08.263517 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:08.264059 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:08.264566 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:08.265371 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:08.265641 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:08.265735 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:08.268020 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:08.268026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:08.268033 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:08.268060 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:08.269367 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:08.269789 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:08.269794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:08.269796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:08.269875 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:08.270174 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:08.270178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:08.270180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:08.270365 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:08.270541 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:08.270779 (+ 238us) server_negotiation.cc:299] Negotiation successful
0621 07:52:08.270892 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260621 07:52:08.274015 18387 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:08.272893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34572 (local address 127.12.12.1:33321)
0621 07:52:08.273038 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:08.273041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:08.273055 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:52:08.273252 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:08.273255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:08.273294 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:08.273417 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:08.273869 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34572: BlockingRecv error: recv got EOF from 127.0.0.1:34572 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260621 07:52:08.274035 12336 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028299065733-12336-0/krb5kdc/krb5cc' of type FILE
Jun 21 07:52:08 dist-test-slave-lbpt krb5kdc[17876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:08.275187 12336 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 07:52:08 dist-test-slave-lbpt krb5kdc[17876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:08.276075 12336 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 07:52:08 dist-test-slave-lbpt krb5kdc[17876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.1@KRBTEST.COM
I20260621 07:52:08.285537 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:08.277001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34580 (local address 127.12.12.1:33321)
0621 07:52:08.277132 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:08.277136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:08.277228 (+ 92us) server_negotiation.cc:407] Connection header received
0621 07:52:08.277386 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:08.277389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:08.277464 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:08.277553 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:08.278086 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:08.278647 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:08.279460 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:08.279680 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:08.279784 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:08.282174 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:08.282178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:08.282180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:08.282203 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:08.283869 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:08.284291 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:08.284296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:08.284298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:08.284393 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:08.284673 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:08.284677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:08.284679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:08.284869 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:08.285007 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:08.285242 (+ 235us) server_negotiation.cc:299] Negotiation successful
0621 07:52:08.285363 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 07:52:08.286916 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 17983
I20260621 07:52:08.292850 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 18117
I20260621 07:52:08.298295 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 18251
I20260621 07:52:08.303735 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 17889
2026-06-21T07:52:08Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2215 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:52:08 dist-test-slave-lbpt krb5kdc[18400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:52:08 dist-test-slave-lbpt krb5kdc[18400](info): set up 2 sockets
Jun 21 07:52:08 dist-test-slave-lbpt krb5kdc[18400](info): commencing operation
krb5kdc: starting...
W20260621 07:52:09.350883 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:52:09 dist-test-slave-lbpt krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:52:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:52:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.62.keytab.
Entry for principal oryx/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.62.keytab.
I20260621 07:52:09.475567 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:39045
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.62.keytab
--principal=oryx/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:39045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:09.591912 18413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:09.592250 18413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:09.592327 18413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:09.596640 18413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:09.596755 18413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:09.596793 18413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:09.596830 18413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:09.596864 18413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:52:09.602757 18413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:39045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.62.keytab
--principal=oryx/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:39045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:09.603992 18413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:09.605118 18413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:09.614446 18418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:09.614456 18419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:09.614463 18421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:09.614641 18413 server_base.cc:1034] running on GCE node
I20260621 07:52:09.615373 18413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:09.616587 18413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:09.617805 18413 hybrid_clock.cc:648] HybridClock initialized: now 1782028329617788 us; error 32 us; skew 500 ppm
Jun 21 07:52:09 dist-test-slave-lbpt krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028329, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:09.620976 18413 init.cc:377] Logged in from keytab as oryx/127.12.12.62@KRBTEST.COM (short username oryx)
I20260621 07:52:09.622156 18413 webserver.cc:460] Webserver started at http://127.12.12.62:36157/ using document root <none> and password file <none>
I20260621 07:52:09.622896 18413 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:09.623004 18413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:09.623270 18413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:09.625631 18413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "bae2da0cc2794cc78af1b0c07edafeaf"
format_stamp: "Formatted at 2026-06-21 07:52:09 on dist-test-slave-lbpt"
server_key: "c3d232281c0472162159f32dbc636348"
server_key_iv: "20ceaf5f80fe31e7a1ab71ae7607e896"
server_key_version: "encryptionkey@0"
I20260621 07:52:09.626266 18413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "bae2da0cc2794cc78af1b0c07edafeaf"
format_stamp: "Formatted at 2026-06-21 07:52:09 on dist-test-slave-lbpt"
server_key: "c3d232281c0472162159f32dbc636348"
server_key_iv: "20ceaf5f80fe31e7a1ab71ae7607e896"
server_key_version: "encryptionkey@0"
I20260621 07:52:09.630643 18413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:52:09.633428 18428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:09.634922 18413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:09.635075 18413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "bae2da0cc2794cc78af1b0c07edafeaf"
format_stamp: "Formatted at 2026-06-21 07:52:09 on dist-test-slave-lbpt"
server_key: "c3d232281c0472162159f32dbc636348"
server_key_iv: "20ceaf5f80fe31e7a1ab71ae7607e896"
server_key_version: "encryptionkey@0"
I20260621 07:52:09.635200 18413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:09.646588 18413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:09.649727 18413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:09.649971 18413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:09.657346 18413 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:39045
I20260621 07:52:09.657634 18480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:39045 every 8 connection(s)
I20260621 07:52:09.658514 18413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:52:09.661617 18481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:09.662308 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 18413
I20260621 07:52:09.662515 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:52:09.662849 12336 external_mini_cluster.cc:1428] Setting key e9f81802362e583c0b73d90796494962
I20260621 07:52:09.668737 18481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf: Bootstrap starting.
I20260621 07:52:09.671435 18481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:09.672197 18481 log.cc:824] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:52:09 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM
I20260621 07:52:09.674677 18481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf: No bootstrap required, opened a new log
I20260621 07:52:09.679009 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:09.664394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.12.12.62:39045)
0621 07:52:09.664857 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:09.664877 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:09.664933 (+ 56us) server_negotiation.cc:407] Connection header received
0621 07:52:09.665746 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:09.665774 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:09.666225 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:09.666654 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:09.667363 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:09.668839 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:09.669787 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:09.670554 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:09.670715 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:09.673350 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:09.673356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:09.673373 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:09.673403 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:09.675461 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:09.675992 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:09.675998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:09.676004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:09.676146 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:09.676499 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:09.676503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:09.676504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:09.677126 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:09.677307 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:09.677649 (+ 342us) server_negotiation.cc:299] Negotiation successful
0621 07:52:09.677970 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":108,"threads_started":1}
I20260621 07:52:09.679388 18481 raft_consensus.cc:348] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } }
I20260621 07:52:09.679651 18481 raft_consensus.cc:374] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:09.679731 18481 raft_consensus.cc:729] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae2da0cc2794cc78af1b0c07edafeaf, State: Initialized, Role: FOLLOWER
I20260621 07:52:09.680239 18481 consensus_queue.cc:260] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } }
I20260621 07:52:09.680403 18481 raft_consensus.cc:388] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:52:09.680500 18481 raft_consensus.cc:482] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:52:09.680599 18481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:09.681612 18481 raft_consensus.cc:504] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } }
I20260621 07:52:09.681978 18481 leader_election.cc:302] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bae2da0cc2794cc78af1b0c07edafeaf; no voters:
I20260621 07:52:09.682327 18481 leader_election.cc:288] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:52:09.682469 18486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:09.682723 18486 raft_consensus.cc:686] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [term 1 LEADER]: Becoming Leader. State: Replica: bae2da0cc2794cc78af1b0c07edafeaf, State: Running, Role: LEADER
I20260621 07:52:09.683039 18486 consensus_queue.cc:237] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } }
I20260621 07:52:09.683596 18481 sys_catalog.cc:564] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:52:09.684770 18487 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } }
I20260621 07:52:09.684912 18487 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:09.684784 18488 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader bae2da0cc2794cc78af1b0c07edafeaf. Latest consensus state: current_term: 1 leader_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } }
I20260621 07:52:09.685600 18488 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:09.686362 18495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:52:09.689113 18495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:52:09.695375 18495 catalog_manager.cc:1347] Generated new cluster ID: d3f80f6fa24041078a598910c09aaaf3
I20260621 07:52:09.695472 18495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:52:09.727252 18495 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:52:09.728616 18495 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:52:09.737517 18495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf: Generated new TSK 0
I20260621 07:52:09.738476 18495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.1.keytab.
Entry for principal oryx/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.1.keytab.
I20260621 07:52:09.771819 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:39045
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.1.keytab
--principal=oryx/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:09.888638 18507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:09.888929 18507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:09.888974 18507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:09.893267 18507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:09.893337 18507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:09.893402 18507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:52:09.898937 18507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.1.keytab
--principal=oryx/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:39045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:09.900116 18507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:09.901193 18507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:09.911144 18513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:09.911103 18515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:09.911094 18512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:09.911521 18507 server_base.cc:1034] running on GCE node
I20260621 07:52:09.911921 18507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:09.912561 18507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:09.913722 18507 hybrid_clock.cc:648] HybridClock initialized: now 1782028329913687 us; error 47 us; skew 500 ppm
Jun 21 07:52:09 dist-test-slave-lbpt krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028329, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:09.916628 18507 init.cc:377] Logged in from keytab as oryx/127.12.12.1@KRBTEST.COM (short username oryx)
I20260621 07:52:09.917661 18507 webserver.cc:460] Webserver started at http://127.12.12.1:44647/ using document root <none> and password file <none>
I20260621 07:52:09.918273 18507 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:09.918366 18507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:09.918637 18507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:09.920729 18507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "85929ce3cf294504a55fd8e9ad3515ba"
format_stamp: "Formatted at 2026-06-21 07:52:09 on dist-test-slave-lbpt"
server_key: "e032c55b3c7276d878cfb0bc458cdea1"
server_key_iv: "da8d3226a715bfd713ddf7f6dec6e811"
server_key_version: "encryptionkey@0"
I20260621 07:52:09.921303 18507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "85929ce3cf294504a55fd8e9ad3515ba"
format_stamp: "Formatted at 2026-06-21 07:52:09 on dist-test-slave-lbpt"
server_key: "e032c55b3c7276d878cfb0bc458cdea1"
server_key_iv: "da8d3226a715bfd713ddf7f6dec6e811"
server_key_version: "encryptionkey@0"
I20260621 07:52:09.926184 18507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:52:09.929020 18522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:09.930371 18507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:52:09.930629 18507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "85929ce3cf294504a55fd8e9ad3515ba"
format_stamp: "Formatted at 2026-06-21 07:52:09 on dist-test-slave-lbpt"
server_key: "e032c55b3c7276d878cfb0bc458cdea1"
server_key_iv: "da8d3226a715bfd713ddf7f6dec6e811"
server_key_version: "encryptionkey@0"
I20260621 07:52:09.930754 18507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:09.942660 18507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:09.945794 18507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:09.946017 18507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:09.946738 18507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:09.947613 18507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:09.947686 18507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:09.947757 18507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:09.947805 18507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:09.956928 18507 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:44609
I20260621 07:52:09.956970 18635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:44609 every 8 connection(s)
I20260621 07:52:09.957912 18507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:52:09.959069 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 18507
I20260621 07:52:09.959200 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:52:09.959545 12336 external_mini_cluster.cc:1428] Setting key ca18ef7116585cf252e59a966fa6f48b
Jun 21 07:52:09 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028329, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.12.12.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.2@KRBTEST.COM" created.
I20260621 07:52:09.973155 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:09.960158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:48343 (local address 127.12.12.62:39045)
0621 07:52:09.960283 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:09.960288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:09.961319 (+ 1031us) server_negotiation.cc:407] Connection header received
0621 07:52:09.962814 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:09.962818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:09.962875 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:09.962963 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:09.964182 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:09.964737 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:09.965891 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:09.966130 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:09.966243 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:09.969110 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:09.969114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:09.969116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:09.969143 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:09.970749 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:09.971444 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:09.971447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:09.971448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:09.971526 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:09.971891 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:09.971897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:09.971901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:09.972200 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:09.972307 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:09.972798 (+ 491us) server_negotiation.cc:299] Negotiation successful
0621 07:52:09.972964 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 07:52:09.974325 18638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:09.960647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.1:48343)
0621 07:52:09.961151 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:09.961206 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:09.962593 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:09.963156 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:09.963164 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:09.963541 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:09.964033 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:09.964046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:09.964879 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:09.964881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:09.965759 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:09.965768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:09.966288 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:09.966292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:09.966527 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:09.967196 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:09.967213 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:09.968924 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:09.970863 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:09.970868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:09.970882 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:09.971337 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:09.971624 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:09.971628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:09.971630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:09.971793 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:09.972304 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:09.972309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:09.972559 (+ 250us) client_negotiation.cc:769] Sending connection context
0621 07:52:09.972722 (+ 163us) client_negotiation.cc:240] Negotiation successful
0621 07:52:09.973108 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
I20260621 07:52:09.975821 18636 heartbeater.cc:347] Connected to a master server at 127.12.12.62:39045
I20260621 07:52:09.976101 18636 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:09.976619 18636 heartbeater.cc:511] Master 127.12.12.62:39045 requested a full tablet report, sending...
I20260621 07:52:09.978199 18445 ts_manager.cc:194] Registered new tserver with Master: 85929ce3cf294504a55fd8e9ad3515ba (127.12.12.1:44609)
I20260621 07:52:09.979442 18445 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:48343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.2.keytab.
Entry for principal oryx/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.2.keytab.
I20260621 07:52:09.987565 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:39045
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.2.keytab
--principal=oryx/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:10.105225 18641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:10.105515 18641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:10.105566 18641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:10.109728 18641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:10.109789 18641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:10.109849 18641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:52:10.115196 18641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.2.keytab
--principal=oryx/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:39045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:10.116199 18641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:10.117161 18641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:10.126708 18646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:10.126730 18649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:10.126730 18647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:10.126785 18641 server_base.cc:1034] running on GCE node
I20260621 07:52:10.127346 18641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:10.128008 18641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:10.129201 18641 hybrid_clock.cc:648] HybridClock initialized: now 1782028330129179 us; error 34 us; skew 500 ppm
Jun 21 07:52:10 dist-test-slave-lbpt krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028330, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:10.132107 18641 init.cc:377] Logged in from keytab as oryx/127.12.12.2@KRBTEST.COM (short username oryx)
I20260621 07:52:10.133059 18641 webserver.cc:460] Webserver started at http://127.12.12.2:37421/ using document root <none> and password file <none>
I20260621 07:52:10.133673 18641 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:10.133770 18641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:10.134034 18641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:10.136178 18641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "1de5e565c961429ab12b8c5e25f72c9f"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "f440182bff9d57962db50fee6c9bbf93"
server_key_iv: "0546e6073d96682cc642e6feb6576e69"
server_key_version: "encryptionkey@0"
I20260621 07:52:10.136745 18641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "1de5e565c961429ab12b8c5e25f72c9f"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "f440182bff9d57962db50fee6c9bbf93"
server_key_iv: "0546e6073d96682cc642e6feb6576e69"
server_key_version: "encryptionkey@0"
I20260621 07:52:10.140547 18641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:52:10.143127 18656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:10.144536 18641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:10.144671 18641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "1de5e565c961429ab12b8c5e25f72c9f"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "f440182bff9d57962db50fee6c9bbf93"
server_key_iv: "0546e6073d96682cc642e6feb6576e69"
server_key_version: "encryptionkey@0"
I20260621 07:52:10.144791 18641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:10.169566 18641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:10.173062 18641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:10.173283 18641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:10.173928 18641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:10.174952 18641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:10.174998 18641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:10.175071 18641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:10.175114 18641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:10.184412 18641 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:42563
I20260621 07:52:10.184566 18769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:42563 every 8 connection(s)
I20260621 07:52:10.185438 18641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:52:10.194753 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 18641
I20260621 07:52:10.194893 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:52:10.195187 12336 external_mini_cluster.cc:1428] Setting key de6a3201d5b77dbc079f25c446b195b9
Jun 21 07:52:10 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028330, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM
I20260621 07:52:10.200496 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.187548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:57441 (local address 127.12.12.62:39045)
0621 07:52:10.187726 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:10.187731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:10.188644 (+ 913us) server_negotiation.cc:407] Connection header received
0621 07:52:10.190013 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:10.190016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:10.190065 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:10.190144 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:10.191412 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.191963 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.193138 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.193409 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.193509 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.196392 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:10.196395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:10.196397 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:10.196423 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:10.198088 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.198740 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.198744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.198746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.198831 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.199214 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.199217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.199218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.199426 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:10.199536 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:10.200226 (+ 690us) server_negotiation.cc:299] Negotiation successful
0621 07:52:10.200369 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 07:52:10.201762 18772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.187996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.2:57441)
0621 07:52:10.188478 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:10.188522 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:10.189783 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:10.190327 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:10.190334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:10.190766 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:10.191267 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.191280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.192086 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.192089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.193001 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.193011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.193556 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.193559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.193703 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.194367 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:10.194385 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:10.196145 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:10.198232 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.198239 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.198256 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.198631 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.198935 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.198938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.198940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.199102 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.199536 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:10.199544 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:10.199903 (+ 359us) client_negotiation.cc:769] Sending connection context
0621 07:52:10.200121 (+ 218us) client_negotiation.cc:240] Negotiation successful
0621 07:52:10.200512 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":106,"threads_started":1}
I20260621 07:52:10.203253 18770 heartbeater.cc:347] Connected to a master server at 127.12.12.62:39045
I20260621 07:52:10.203593 18770 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:10.204160 18770 heartbeater.cc:511] Master 127.12.12.62:39045 requested a full tablet report, sending...
I20260621 07:52:10.205385 18445 ts_manager.cc:194] Registered new tserver with Master: 1de5e565c961429ab12b8c5e25f72c9f (127.12.12.2:42563)
WARNING: no policy specified for oryx/127.12.12.3@KRBTEST.COM; defaulting to no policy
I20260621 07:52:10.206092 18445 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:57441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.3.keytab.
Entry for principal oryx/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.3.keytab.
I20260621 07:52:10.225096 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:39045
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.3.keytab
--principal=oryx/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:10.340488 18775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:10.340792 18775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:10.340837 18775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:10.344923 18775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:10.345001 18775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:10.345062 18775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:52:10.350773 18775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.3.keytab
--principal=oryx/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:39045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:10.351827 18775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:10.352833 18775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:10.362900 18781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:10.362900 18780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:10.362941 18775 server_base.cc:1034] running on GCE node
W20260621 07:52:10.362902 18783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:10.363601 18775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:10.364233 18775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:10.365409 18775 hybrid_clock.cc:648] HybridClock initialized: now 1782028330365384 us; error 36 us; skew 500 ppm
Jun 21 07:52:10 dist-test-slave-lbpt krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028330, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:10.368362 18775 init.cc:377] Logged in from keytab as oryx/127.12.12.3@KRBTEST.COM (short username oryx)
I20260621 07:52:10.369280 18775 webserver.cc:460] Webserver started at http://127.12.12.3:46713/ using document root <none> and password file <none>
I20260621 07:52:10.369833 18775 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:10.369894 18775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:10.370060 18775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:10.372184 18775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "5906911d01134202bed9be4d170b2875"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "8442aeb7740b735c7487275790bd6e16"
server_key_iv: "7b00b653f686a315c4859a6080c65ace"
server_key_version: "encryptionkey@0"
I20260621 07:52:10.372705 18775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "5906911d01134202bed9be4d170b2875"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "8442aeb7740b735c7487275790bd6e16"
server_key_iv: "7b00b653f686a315c4859a6080c65ace"
server_key_version: "encryptionkey@0"
I20260621 07:52:10.376588 18775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260621 07:52:10.379273 18790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:10.380797 18775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:10.380923 18775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "5906911d01134202bed9be4d170b2875"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "8442aeb7740b735c7487275790bd6e16"
server_key_iv: "7b00b653f686a315c4859a6080c65ace"
server_key_version: "encryptionkey@0"
I20260621 07:52:10.381026 18775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:10.400616 18775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:10.404201 18775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:10.404434 18775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:10.405057 18775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:10.405906 18775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:10.405984 18775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:10.406059 18775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:10.406101 18775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:10.415246 18775 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:42983
I20260621 07:52:10.415335 18903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:42983 every 8 connection(s)
I20260621 07:52:10.416206 18775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:52:10.422042 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 18775
I20260621 07:52:10.422204 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:52:10.422600 12336 external_mini_cluster.cc:1428] Setting key ae68849d5e2159765ead0d7dba97443c
Jun 21 07:52:10 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028330, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM
I20260621 07:52:10.430866 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.418090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:36637 (local address 127.12.12.62:39045)
0621 07:52:10.418236 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:10.418239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:10.419171 (+ 932us) server_negotiation.cc:407] Connection header received
0621 07:52:10.420613 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:10.420616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:10.420677 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:10.420789 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:10.422054 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.422828 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.424039 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.424278 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.424390 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.426960 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:10.426963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:10.426965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:10.426990 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:10.428455 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.429165 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.429170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.429174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.429279 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.429713 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.429716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.429717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.429917 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:10.430014 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:10.430623 (+ 609us) server_negotiation.cc:299] Negotiation successful
0621 07:52:10.430738 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 07:52:10.431926 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.418558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.3:36637)
0621 07:52:10.419016 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:10.419062 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:10.420369 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:10.420977 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:10.420985 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:10.421372 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:10.421903 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.421917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.422977 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.422979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.423908 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.423918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.424441 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.424446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.424611 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.425137 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:10.425152 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:10.426783 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:10.428586 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.428591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.428609 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.428893 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.429420 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.429423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.429424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.429594 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.430026 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:10.430031 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:10.430280 (+ 249us) client_negotiation.cc:769] Sending connection context
0621 07:52:10.430566 (+ 286us) client_negotiation.cc:240] Negotiation successful
0621 07:52:10.430879 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20260621 07:52:10.433230 18904 heartbeater.cc:347] Connected to a master server at 127.12.12.62:39045
I20260621 07:52:10.433495 18904 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:10.433971 18904 heartbeater.cc:511] Master 127.12.12.62:39045 requested a full tablet report, sending...
I20260621 07:52:10.435087 18445 ts_manager.cc:194] Registered new tserver with Master: 5906911d01134202bed9be4d170b2875 (127.12.12.3:42983)
I20260621 07:52:10.435770 18445 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:36637
I20260621 07:52:10.436578 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.12.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.61.keytab.
Entry for principal oryx/127.12.12.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.61.keytab.
I20260621 07:52:10.463661 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.61:35893
--webserver_interface=127.12.12.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.61.keytab
--principal=oryx/127.12.12.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:39045,127.12.12.61:35893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:10.578965 18909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:10.579262 18909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:10.579305 18909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:10.583495 18909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:10.583567 18909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:10.583621 18909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:10.583662 18909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:10.583688 18909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:52:10.589562 18909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:39045,127.12.12.61:35893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.61.keytab
--principal=oryx/127.12.12.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.61:35893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.12.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
W20260621 07:52:10.589792 18909 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.12.62:39045,127.12.12.61:35893'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260621 07:52:10.590816 18909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:10.591848 18909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:10.600462 18917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:10.600462 18914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:10.600543 18915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:10.600838 18909 server_base.cc:1034] running on GCE node
I20260621 07:52:10.601500 18909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:10.602756 18909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:10.603945 18909 hybrid_clock.cc:648] HybridClock initialized: now 1782028330603911 us; error 48 us; skew 500 ppm
Jun 21 07:52:10 dist-test-slave-lbpt krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028330, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:10.606814 18909 init.cc:377] Logged in from keytab as oryx/127.12.12.61@KRBTEST.COM (short username oryx)
I20260621 07:52:10.607780 18909 webserver.cc:460] Webserver started at http://127.12.12.61:39483/ using document root <none> and password file <none>
I20260621 07:52:10.608335 18909 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:10.608397 18909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:10.608572 18909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:10.610806 18909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/data/instance:
uuid: "2aeaf63573984aafa402286856776296"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "c4c4760f0831e30246a891d9a3d5d863"
server_key_iv: "948568662b1adce33969349e664e3893"
server_key_version: "encryptionkey@0"
I20260621 07:52:10.611323 18909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/wal/instance:
uuid: "2aeaf63573984aafa402286856776296"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "c4c4760f0831e30246a891d9a3d5d863"
server_key_iv: "948568662b1adce33969349e664e3893"
server_key_version: "encryptionkey@0"
I20260621 07:52:10.615235 18909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:52:10.617792 18924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:10.619261 18909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:10.619410 18909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/wal
uuid: "2aeaf63573984aafa402286856776296"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "c4c4760f0831e30246a891d9a3d5d863"
server_key_iv: "948568662b1adce33969349e664e3893"
server_key_version: "encryptionkey@0"
I20260621 07:52:10.619534 18909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:10.636992 18909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:10.640451 18909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:10.640671 18909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:10.648010 18909 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.61:35893
I20260621 07:52:10.648310 18976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.61:35893 every 8 connection(s)
I20260621 07:52:10.649183 18909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/data/info.pb
I20260621 07:52:10.650537 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 18909
I20260621 07:52:10.652563 18977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:10.657101 18977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:52:10 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.12.61@KRBTEST.COM
Jun 21 07:52:10 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028330, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.61@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM
I20260621 07:52:10.667219 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.652057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.12.12.61:35893)
0621 07:52:10.652470 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:10.652489 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:10.652532 (+ 43us) server_negotiation.cc:407] Connection header received
0621 07:52:10.653366 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:10.653388 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:10.653794 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:10.654167 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:10.655005 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.656506 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.657613 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.658297 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.658484 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.661403 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:10.661409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:10.661420 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:10.661450 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:10.664143 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.664625 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.664631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.664636 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.664782 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.665067 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.665071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.665072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.665508 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:10.665726 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:10.665942 (+ 216us) server_negotiation.cc:299] Negotiation successful
0621 07:52:10.666224 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":108,"threads_started":1}
I20260621 07:52:10.669170 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.658897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:49254)
0621 07:52:10.659586 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:10.659637 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:10.659737 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:10.660034 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:10.660039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:10.660464 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:10.660607 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.660613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.661624 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.661627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.662529 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.662541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.663078 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.663080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.663159 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.663642 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:10.663653 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:10.665084 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:10.667428 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.667436 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.667443 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.667707 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.668005 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.668007 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.668009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.668081 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.668545 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:10.668551 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:10.668658 (+ 107us) client_negotiation.cc:769] Sending connection context
0621 07:52:10.668797 (+ 139us) client_negotiation.cc:240] Negotiation successful
0621 07:52:10.668981 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":141,"threads_started":1}
I20260621 07:52:10.669169 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.659319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.12.12.62:39045)
0621 07:52:10.659755 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:10.659760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:10.659779 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:52:10.659826 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:10.659829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:10.659886 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:10.659981 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:10.660739 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.661489 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.662634 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.662968 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.663073 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.665214 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:10.665218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:10.665221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:10.665247 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:10.667310 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.667814 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.667817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.667819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.667906 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.668175 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.668178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.668180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.668406 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:10.668472 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:10.668880 (+ 408us) server_negotiation.cc:299] Negotiation successful
0621 07:52:10.669001 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330}
I20260621 07:52:10.670470 18977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:52:10 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028330, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.61@KRBTEST.COM for oryx/127.12.12.61@KRBTEST.COM
I20260621 07:52:10.678503 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.670903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:35893 (local address 127.0.0.1:48474)
0621 07:52:10.671018 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:10.671032 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:10.671108 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:10.671368 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:10.671370 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:10.671523 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:10.671611 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.671616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.672955 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.672957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.673486 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.673494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.673939 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.673942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.674006 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.674526 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:10.674539 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:10.675708 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:10.677225 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.677227 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.677228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.677402 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.677652 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.677654 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.677655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.677724 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.678047 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:10.678049 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:10.678120 (+ 71us) client_negotiation.cc:769] Sending connection context
0621 07:52:10.678202 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 07:52:10.678297 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260621 07:52:10.678614 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.671062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48474 (local address 127.12.12.61:35893)
0621 07:52:10.671176 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:10.671180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:10.671191 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:52:10.671225 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:10.671228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:10.671266 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:10.671346 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:10.671707 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.672828 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.673607 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.673832 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.673930 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.675823 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:10.675826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:10.675828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:10.675849 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:10.677127 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.677496 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.677499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.677500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.677572 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.677797 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.677799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.677800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.677969 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:10.678043 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:10.678276 (+ 233us) server_negotiation.cc:299] Negotiation successful
0621 07:52:10.678369 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:52:10.681298 18977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: Bootstrap starting.
I20260621 07:52:10.683622 18977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:10.684456 18977 log.cc:824] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:10.686507 18977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: No bootstrap required, opened a new log
I20260621 07:52:10.689518 18977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } }
I20260621 07:52:10.689774 18977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:10.689827 18977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aeaf63573984aafa402286856776296, State: Initialized, Role: FOLLOWER
I20260621 07:52:10.690274 18977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } }
I20260621 07:52:10.690800 18983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } } }
I20260621 07:52:10.690941 18983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:52:10.691563 18977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: configured and running, proceeding with master startup.
W20260621 07:52:10.696262 18994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:52:10.696346 18994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:52:10.698112 18909 master_runner.cc:428] Detected that this master 2aeaf63573984aafa402286856776296 is joining an existing cluster
I20260621 07:52:10.698203 18909 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.12.61:35893
I20260621 07:52:10.708880 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.700390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.12.12.61:35893)
0621 07:52:10.700515 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:10.700520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:10.700999 (+ 479us) server_negotiation.cc:407] Connection header received
0621 07:52:10.701146 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:10.701150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:10.701194 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:10.701256 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:10.701753 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.702252 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.703291 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.703531 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.703642 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.705320 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:10.705324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:10.705326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:10.705348 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:10.707030 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.707538 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.707543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.707545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.707623 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.707928 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.707932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.707935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.708157 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:10.708266 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:10.708540 (+ 274us) server_negotiation.cc:299] Negotiation successful
0621 07:52:10.708692 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":67}
I20260621 07:52:10.708894 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.700617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:35893 (local address 127.0.0.1:48486)
0621 07:52:10.700910 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:10.700932 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:10.701054 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:10.701353 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:10.701355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:10.701543 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:10.701638 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.701643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.702366 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.702369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.703134 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.703184 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.703651 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.703654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.703839 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.704641 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:10.704652 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:10.705184 (+ 532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:10.707143 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.707149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.707154 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.707424 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.707724 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.707728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.707731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.707815 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.708296 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:10.708298 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:10.708382 (+ 84us) client_negotiation.cc:769] Sending connection context
0621 07:52:10.708498 (+ 116us) client_negotiation.cc:240] Negotiation successful
0621 07:52:10.708643 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":19,"thread_start_us":108,"threads_started":1}
I20260621 07:52:10.709884 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.700368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49262 (local address 127.12.12.62:39045)
0621 07:52:10.700516 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:10.700521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:10.700832 (+ 311us) server_negotiation.cc:407] Connection header received
0621 07:52:10.701030 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:10.701033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:10.701087 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:10.701186 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:10.701775 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.702614 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.703923 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.704208 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.704333 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.705458 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:10.705462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:10.705465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:10.705495 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:10.707805 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.708562 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.708566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.708568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.708674 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.709000 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.709003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.709005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.709211 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:10.709278 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:10.709620 (+ 342us) server_negotiation.cc:299] Negotiation successful
0621 07:52:10.709735 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:52:10.710510 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.700440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:49262)
0621 07:52:10.700746 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:10.700764 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:10.700917 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:10.701215 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:10.701218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:10.701479 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:10.701631 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.701639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.702754 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.702757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.703813 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.703824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.704351 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.704355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.704449 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.704884 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:10.704897 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:10.705319 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:10.707937 (+ 2618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.707940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.707942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.708447 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.708788 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.708791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.708793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.708889 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.709351 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:10.709356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:10.709444 (+ 88us) client_negotiation.cc:769] Sending connection context
0621 07:52:10.710156 (+ 712us) client_negotiation.cc:240] Negotiation successful
0621 07:52:10.710313 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":74,"thread_start_us":109,"threads_started":1}
Jun 21 07:52:10 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028329, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.62@KRBTEST.COM for oryx/127.12.12.61@KRBTEST.COM
I20260621 07:52:10.724390 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.714590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.12.12.61:35893)
0621 07:52:10.714705 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:10.714709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:10.715069 (+ 360us) server_negotiation.cc:407] Connection header received
0621 07:52:10.715279 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:10.715282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:10.715331 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:10.715428 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:10.716070 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.716568 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.717454 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.717681 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.717785 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.720290 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:10.720293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:10.720296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:10.720319 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:10.722064 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.722686 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.722690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.722692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.722776 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.723109 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.723116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.723118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.723419 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:10.723556 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:10.724106 (+ 550us) server_negotiation.cc:299] Negotiation successful
0621 07:52:10.724221 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:52:10.724408 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.714576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:35893 (local address 127.0.0.1:48496)
0621 07:52:10.714946 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:10.714989 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:10.715143 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:10.715461 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:10.715467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:10.715840 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:10.715953 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.715960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.716675 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.716680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.717327 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.717336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.717789 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.717793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.717919 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.718483 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:10.718497 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:10.720158 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:10.722182 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.722192 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.722199 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.722530 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.722883 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.722886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.722888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.723004 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.723619 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:10.723628 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:10.723871 (+ 243us) client_negotiation.cc:769] Sending connection context
0621 07:52:10.724037 (+ 166us) client_negotiation.cc:240] Negotiation successful
0621 07:52:10.724237 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":107,"threads_started":1}
I20260621 07:52:10.726316 18445 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.12.61:35893: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2aeaf63573984aafa402286856776296"
member_type: NON_VOTER
last_known_addr {
host: "127.12.12.61"
port: 35893
}
attrs {
promote: true
}
}
dest_uuid: "bae2da0cc2794cc78af1b0c07edafeaf"
cas_config_opid_index: -1
I20260621 07:52:10.727010 18445 consensus_queue.cc:237] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: NON_VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: true } }
I20260621 07:52:10.728688 19004 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2aeaf63573984aafa402286856776296 (127.12.12.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: NON_VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: true } } }
I20260621 07:52:10.729657 18951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 0 FOLLOWER]: Advancing to term 1
WARNING: no policy specified for oryx/127.12.12.60@KRBTEST.COM; defaulting to no policy
I20260621 07:52:10.732090 18951 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 FOLLOWER]: Refusing update from remote peer bae2da0cc2794cc78af1b0c07edafeaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260621 07:52:10.732393 19004 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.12.61:35893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.60@KRBTEST.COM" created.
I20260621 07:52:10.732811 19005 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: NON_VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: true } } }
I20260621 07:52:10.733011 19005 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:10.733114 18909 master.cc:537] Master@127.12.12.61:35893 shutting down...
I20260621 07:52:10.733268 19005 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: NON_VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.002s
W20260621 07:52:10.737670 18432 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260621 07:52:10.738356 18909 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:10.738677 18909 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:10.738826 18909 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: stopping tablet replica
W20260621 07:52:10.739462 18432 consensus_peers.cc:487] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf -> Peer 2aeaf63573984aafa402286856776296 (127.12.12.61:35893): Couldn't send request to peer 2aeaf63573984aafa402286856776296. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.60.keytab.
Entry for principal oryx/127.12.12.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.60.keytab.
I20260621 07:52:10.750115 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.60:44397
--webserver_interface=127.12.12.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.60.keytab
--principal=oryx/127.12.12.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:39045,127.12.12.61:35893,127.12.12.60:44397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:52:10.754937 18909 master.cc:552] Master@127.12.12.61:35893 shutdown complete.
I20260621 07:52:10.755023 18909 master_runner.cc:305] Clearing existing system tablet
I20260621 07:52:10.755321 18909 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:10.755391 18909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:10.757050 18909 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 07:52:10.758116 19011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:10.758390 18909 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:10.758643 18909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 07:52:10.758694 18909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/wal
uuid: "2aeaf63573984aafa402286856776296"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "c4c4760f0831e30246a891d9a3d5d863"
server_key_iv: "948568662b1adce33969349e664e3893"
server_key_version: "encryptionkey@0"
I20260621 07:52:10.762281 18909 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:10.764720 18909 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 07:52:10.764814 18909 log.cc:1196] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260621 07:52:10.765076 18909 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: Deleting consensus metadata
I20260621 07:52:10.765240 18909 master_runner.cc:315] Copying system tablet from 127.12.12.62:39045
I20260621 07:52:10.766433 18909 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: tablet copy: Beginning tablet copy session from remote peer at address 127.12.12.62:39045
I20260621 07:52:10.779114 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.767121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.12.12.62:39045)
0621 07:52:10.767247 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:10.767253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:10.770799 (+ 3546us) server_negotiation.cc:407] Connection header received
0621 07:52:10.770935 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:10.770941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:10.771014 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:10.771124 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:10.771655 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.772506 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.773734 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.774075 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.774186 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.775582 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:10.775586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:10.775588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:10.775617 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:10.777285 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.777766 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.777769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.777771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.777869 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.778155 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.778158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.778159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.778379 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:10.778542 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:10.778838 (+ 296us) server_negotiation.cc:299] Negotiation successful
0621 07:52:10.778958 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:52:10.779143 19017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.767008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:49274)
0621 07:52:10.770664 (+ 3656us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:10.770683 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:10.770828 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:10.771161 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:10.771165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:10.771390 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:10.771531 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.771538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.772632 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.772636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.773590 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.773602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.774208 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.774212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.774313 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.774949 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:10.774964 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:10.775420 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:10.777401 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.777404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.777406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.777662 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.777968 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.777971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.777973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.778064 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.778546 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:10.778549 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:10.778651 (+ 102us) client_negotiation.cc:769] Sending connection context
0621 07:52:10.778762 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 07:52:10.778910 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3565,"thread_start_us":96,"threads_started":1}
I20260621 07:52:10.779971 18465 tablet_copy_service.cc:140] P bae2da0cc2794cc78af1b0c07edafeaf: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2aeaf63573984aafa402286856776296 ({username='oryx', principal='oryx/127.12.12.61@KRBTEST.COM'} at 127.0.0.1:49274)
I20260621 07:52:10.780154 18465 tablet_copy_service.cc:161] P bae2da0cc2794cc78af1b0c07edafeaf: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2aeaf63573984aafa402286856776296 at {username='oryx', principal='oryx/127.12.12.61@KRBTEST.COM'} at 127.0.0.1:49274: session id = 2aeaf63573984aafa402286856776296-00000000000000000000000000000000
I20260621 07:52:10.781844 18465 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 07:52:10.783514 18909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 07:52:10.786221 18909 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: tablet copy: Starting download of 0 data blocks...
I20260621 07:52:10.786441 18909 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: tablet copy: Starting download of 1 WAL segments...
I20260621 07:52:10.788246 18909 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 07:52:10.790395 18465 tablet_copy_service.cc:342] P bae2da0cc2794cc78af1b0c07edafeaf: Request end of tablet copy session 2aeaf63573984aafa402286856776296-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.12.61@KRBTEST.COM'} at 127.0.0.1:49274
I20260621 07:52:10.790578 18465 tablet_copy_service.cc:433] P bae2da0cc2794cc78af1b0c07edafeaf: ending tablet copy session 2aeaf63573984aafa402286856776296-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2aeaf63573984aafa402286856776296
I20260621 07:52:10.798372 18909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:10.800264 19021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:10.800316 19018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:10.800850 18909 server_base.cc:1034] running on GCE node
W20260621 07:52:10.802850 19019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:10.803092 18909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:10.803438 18909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:10.804563 18909 hybrid_clock.cc:648] HybridClock initialized: now 1782028330804541 us; error 41 us; skew 500 ppm
Jun 21 07:52:10 dist-test-slave-lbpt krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028330, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:10.806130 18909 init.cc:377] Logged in from keytab as oryx/127.12.12.61@KRBTEST.COM (short username oryx)
I20260621 07:52:10.806617 18909 webserver.cc:460] Webserver started at http://127.12.12.61:39531/ using document root <none> and password file <none>
I20260621 07:52:10.806816 18909 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:10.806860 18909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:10.808177 18909 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 07:52:10.809208 19028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:10.809453 18909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 07:52:10.809557 18909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/wal
uuid: "2aeaf63573984aafa402286856776296"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "c4c4760f0831e30246a891d9a3d5d863"
server_key_iv: "948568662b1adce33969349e664e3893"
server_key_version: "encryptionkey@0"
I20260621 07:52:10.809679 18909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:10.837054 18909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:10.840112 18909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:10.845479 18909 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.61:35893
I20260621 07:52:10.846223 18909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-1/data/info.pb
I20260621 07:52:10.845502 19080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.61:35893 every 8 connection(s)
I20260621 07:52:10.848210 19081 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:52:10.849391 19081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: Bootstrap starting.
I20260621 07:52:10.865547 19081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:52:10.865878 19081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: Bootstrap complete.
I20260621 07:52:10.866525 19081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: NON_VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: true } }
I20260621 07:52:10.866662 19081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2aeaf63573984aafa402286856776296, State: Initialized, Role: LEARNER
I20260621 07:52:10.866770 19081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: NON_VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: true } }
I20260621 07:52:10.867249 19081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:52:10.870332 19084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: NON_VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: true } } }
I20260621 07:52:10.870476 19084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: This master's current role is: LEARNER
I20260621 07:52:10.872097 19095 catalog_manager.cc:1259] Loaded cluster ID: d3f80f6fa24041078a598910c09aaaf3
I20260621 07:52:10.872155 19095 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: loading cluster ID for follower catalog manager: success
I20260621 07:52:10.874241 19095 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: acquiring CA information for follower catalog manager: success
I20260621 07:52:10.875702 19095 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 07:52:10.908092 19009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:10.908420 19009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:10.908502 19009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:10.913066 19009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:10.913136 19009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:10.913156 19009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:10.913172 19009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:10.913188 19009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:52:10.918901 19009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:39045,127.12.12.61:35893,127.12.12.60:44397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.60.keytab
--principal=oryx/127.12.12.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.60:44397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.12.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:10.920001 19009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:10.921049 19009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:10.929853 19103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:10.929857 19100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:10.929847 19101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:10.930157 19009 server_base.cc:1034] running on GCE node
I20260621 07:52:10.930857 19009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:10.932080 19009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:10.933298 19009 hybrid_clock.cc:648] HybridClock initialized: now 1782028330933286 us; error 41 us; skew 500 ppm
Jun 21 07:52:10 dist-test-slave-lbpt krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028330, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:10.936712 19009 init.cc:377] Logged in from keytab as oryx/127.12.12.60@KRBTEST.COM (short username oryx)
I20260621 07:52:10.937841 19009 webserver.cc:460] Webserver started at http://127.12.12.60:45933/ using document root <none> and password file <none>
I20260621 07:52:10.938524 19009 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:10.938621 19009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:10.938959 19009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:10.941166 19009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/data/instance:
uuid: "d8699395d82d49d3a6312d71a6287a1c"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "c102c765caefc73b25bfc12542ac81b2"
server_key_iv: "ba51f9c25dfe5d2b6a9601a8d7f5d2d6"
server_key_version: "encryptionkey@0"
I20260621 07:52:10.941759 19009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/wal/instance:
uuid: "d8699395d82d49d3a6312d71a6287a1c"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "c102c765caefc73b25bfc12542ac81b2"
server_key_iv: "ba51f9c25dfe5d2b6a9601a8d7f5d2d6"
server_key_version: "encryptionkey@0"
I20260621 07:52:10.945823 19009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 07:52:10.948623 19110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:10.950124 19009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 07:52:10.950273 19009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/wal
uuid: "d8699395d82d49d3a6312d71a6287a1c"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "c102c765caefc73b25bfc12542ac81b2"
server_key_iv: "ba51f9c25dfe5d2b6a9601a8d7f5d2d6"
server_key_version: "encryptionkey@0"
I20260621 07:52:10.950412 19009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:10.964432 19009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:10.968272 19009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:10.968489 19009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:10.975963 19009 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.60:44397
I20260621 07:52:10.976255 19162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.60:44397 every 8 connection(s)
I20260621 07:52:10.977106 19009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/data/info.pb
I20260621 07:52:10.978396 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 19009
I20260621 07:52:10.980171 19163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:10.982086 18636 heartbeater.cc:503] Master 127.12.12.62:39045 was elected leader, sending a full tablet report...
I20260621 07:52:10.984972 19163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:52:10 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.12.60@KRBTEST.COM
Jun 21 07:52:10 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028330, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.60@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM
I20260621 07:52:10.994830 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.979958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.12.12.60:44397)
0621 07:52:10.980357 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:10.980384 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:10.980419 (+ 35us) server_negotiation.cc:407] Connection header received
0621 07:52:10.981078 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:10.981094 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:10.981425 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:10.981766 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:10.982734 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.984022 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.985105 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.985804 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.985981 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.989105 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:10.989111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:10.989123 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:10.989154 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:10.991188 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.991737 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.991742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.991746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.991890 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.992230 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.992233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.992234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.992695 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:10.992875 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:10.993241 (+ 366us) server_negotiation.cc:299] Negotiation successful
0621 07:52:10.993566 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":140,"threads_started":1}
I20260621 07:52:10.996840 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.986778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:49278)
0621 07:52:10.987123 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:10.987172 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:10.987291 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:10.987614 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:10.987618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:10.987968 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:10.988089 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.988095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.988953 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.988956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.990010 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.990024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.990676 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.990679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:10.990769 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.991315 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:10.991332 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:10.993031 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:10.995095 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.995102 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.995108 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.995398 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.995684 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:10.995687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:10.995689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:10.995777 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:10.996202 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:10.996208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:10.996325 (+ 117us) client_negotiation.cc:769] Sending connection context
0621 07:52:10.996488 (+ 163us) client_negotiation.cc:240] Negotiation successful
0621 07:52:10.996665 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":110,"threads_started":1}
I20260621 07:52:10.996874 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.986890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.12.12.62:39045)
0621 07:52:10.987038 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:10.987044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:10.987261 (+ 217us) server_negotiation.cc:407] Connection header received
0621 07:52:10.987392 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:10.987398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:10.987461 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:10.987564 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:10.988214 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.988828 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.990120 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:10.990558 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:10.990676 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:10.993175 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:10.993178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:10.993192 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:10.993221 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:10.994947 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.995510 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.995513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.995514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.995592 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:10.995908 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:10.995911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:10.995912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:10.996115 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:10.996202 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:10.996546 (+ 344us) server_negotiation.cc:299] Negotiation successful
0621 07:52:10.996649 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 07:52:10.998303 19163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:52:10 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028330, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.60@KRBTEST.COM for oryx/127.12.12.61@KRBTEST.COM
I20260621 07:52:11.006927 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.998857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:35893 (local address 127.0.0.1:48500)
0621 07:52:10.998997 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:10.999011 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:10.999080 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:10.999482 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:10.999485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:10.999657 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:10.999744 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:10.999749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.000659 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.000662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.001204 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.001211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.001680 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.001683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.001776 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.002232 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:11.002243 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:11.003558 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:11.005278 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.005281 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.005283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.005508 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.005771 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.005773 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.005774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.005871 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.006293 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:11.006296 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:11.006404 (+ 108us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.006596 (+ 192us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.006733 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 07:52:11.006951 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:10.998985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48500 (local address 127.12.12.61:35893)
0621 07:52:10.999256 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:10.999260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:10.999271 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:52:10.999305 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:10.999307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:10.999343 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:10.999472 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:10.999861 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.000541 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.001330 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.001570 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.001683 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.003716 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.003719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.003721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.003746 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.005164 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.005605 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.005608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.005609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.005690 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.005948 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.005950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.005952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.006127 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.006201 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.006691 (+ 490us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.006809 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":86,"threads_started":1}
I20260621 07:52:11.008033 19163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:52:11 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028330, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.60@KRBTEST.COM for oryx/127.12.12.60@KRBTEST.COM
I20260621 07:52:11.016328 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.008653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.12.12.60:44397)
0621 07:52:11.008765 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.008770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.008842 (+ 72us) server_negotiation.cc:407] Connection header received
0621 07:52:11.008987 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.008990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.009030 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.009109 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.009461 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.010342 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.011226 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.011470 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.011628 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.013412 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.013415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.013418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.013448 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.014812 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.015201 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.015207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.015209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.015279 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.015537 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.015543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.015545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.015711 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.015811 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.016054 (+ 243us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.016193 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":38}
I20260621 07:52:11.016328 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.008615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:44397 (local address 127.0.0.1:55832)
0621 07:52:11.008765 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.008780 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.008871 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.009123 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.009125 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.009292 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:11.009354 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.009358 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.010558 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.010563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.011111 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.011120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.011576 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.011579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.011674 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.012124 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:11.012134 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:11.013288 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:11.014913 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.014920 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.014929 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.015109 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.015371 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.015374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.015376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.015453 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.015811 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:11.015814 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:11.015896 (+ 82us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.016002 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.016143 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260621 07:52:11.019543 19163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: Bootstrap starting.
I20260621 07:52:11.021847 19163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:11.022668 19163 log.cc:824] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:11.024727 19163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: No bootstrap required, opened a new log
I20260621 07:52:11.027809 19163 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } }
I20260621 07:52:11.028045 19163 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:11.028128 19163 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8699395d82d49d3a6312d71a6287a1c, State: Initialized, Role: FOLLOWER
I20260621 07:52:11.028581 19163 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } }
I20260621 07:52:11.029081 19169 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } } }
I20260621 07:52:11.029197 19169 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:52:11.029799 19163 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [sys.catalog]: configured and running, proceeding with master startup.
W20260621 07:52:11.034636 19180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:52:11.034720 19180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:52:11.037766 19009 master_runner.cc:428] Detected that this master d8699395d82d49d3a6312d71a6287a1c is joining an existing cluster
I20260621 07:52:11.037850 19009 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.12.60:44397
I20260621 07:52:11.048749 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 18507
I20260621 07:52:11.050870 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.040158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.12.12.61:35893)
0621 07:52:11.040276 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.040281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.040993 (+ 712us) server_negotiation.cc:407] Connection header received
0621 07:52:11.041148 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.041151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.041199 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.041252 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.041855 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.042626 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.044299 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.044646 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.045093 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.046635 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.046639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.046641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.046671 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.048408 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.049387 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.049391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.049393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.049502 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.049843 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.049846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.049848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.050084 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.050177 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.050572 (+ 395us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.050704 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:52:11.051077 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.040068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49294 (local address 127.12.12.62:39045)
0621 07:52:11.040200 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.040207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.040757 (+ 550us) server_negotiation.cc:407] Connection header received
0621 07:52:11.040923 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.040928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.040986 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.041038 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.041730 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.042571 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.043953 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.044301 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.044412 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.045711 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.045715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.045717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.045743 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.047520 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.048098 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.048105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.048110 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.048206 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.049879 (+ 1673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.049884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.049889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.050150 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.050230 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.050696 (+ 466us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.050840 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:52:11.051777 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.040298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:49294)
0621 07:52:11.040685 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.040696 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.040816 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.041186 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.041189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.041471 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:11.041593 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.041600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.042711 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.042714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.043777 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.043787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.044421 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.044424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.044515 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.045077 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:11.045091 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:11.045581 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:11.047637 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.047640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.047642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.047905 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.048333 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.048339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.048342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.048443 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.050321 (+ 1878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:11.050327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:11.050492 (+ 165us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.051226 (+ 734us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.051469 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":28,"thread_start_us":78,"threads_started":1}
I20260621 07:52:11.051800 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.040380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:35893 (local address 127.0.0.1:48516)
0621 07:52:11.040918 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.040931 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.041046 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.041391 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.041393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.041587 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:11.041736 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.041743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.042745 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.042748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.044188 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.044192 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.045218 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.045221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.045324 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.046040 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:11.046051 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:11.046519 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:11.048532 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.048534 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.048537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.048754 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.049630 (+ 876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.049636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.049641 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.049742 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.050221 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:11.050224 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:11.050346 (+ 122us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.051467 (+ 1121us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.051617 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":29,"thread_start_us":85,"threads_started":1}
I20260621 07:52:11.057247 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:44609
--local_ip_for_outbound_sockets=127.12.12.1
--tserver_master_addrs=127.12.12.62:39045
--webserver_port=44647
--webserver_interface=127.12.12.1
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.1.keytab
--principal=oryx/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.12.62:39045,127.12.12.61:35893
--tserver_master_addrs=127.12.12.62:39045,127.12.12.61:35893,127.12.12.60:44397 with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:52:11 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028329, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.62@KRBTEST.COM for oryx/127.12.12.60@KRBTEST.COM
I20260621 07:52:11.067890 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.040063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:44397 (local address 127.0.0.1:55846)
0621 07:52:11.040568 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.040582 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.040686 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.041026 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.041029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.041307 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:11.041447 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.041455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.042511 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.042514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.043479 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.043489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.044535 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.044539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.044630 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.063087 (+ 18457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:11.063108 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:11.063663 (+ 555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:11.065897 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.065900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.065902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.066142 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.066507 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.066510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.066512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.066609 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.067040 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:11.067042 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:11.067130 (+ 88us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.067508 (+ 378us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.067684 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":27,"thread_start_us":124,"threads_started":1}
I20260621 07:52:11.068652 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.040450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.12.12.60:44397)
0621 07:52:11.040548 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.040552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.040670 (+ 118us) server_negotiation.cc:407] Connection header received
0621 07:52:11.040784 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.040787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.040834 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.040914 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.041566 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.042364 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.043601 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.043987 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.044056 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.063821 (+ 19765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.063825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.063828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.063855 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.065783 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.066236 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.066239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.066241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.066307 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.066701 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.066705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.066706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.066894 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.066989 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.068372 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.068489 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":40}
I20260621 07:52:11.072022 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.059400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:44397 (local address 127.0.0.1:55860)
0621 07:52:11.059757 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.059775 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.059886 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.060216 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.060219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.060430 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:11.060545 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.060552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.061713 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.061717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.062678 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.062690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.064329 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.064336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.064495 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.064990 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:11.065010 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:11.066744 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:11.070391 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.070394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.070396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.070674 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.070986 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.070989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.070991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.071085 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.071504 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:11.071507 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:11.071610 (+ 103us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.071709 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.071854 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263}
I20260621 07:52:11.072085 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.059487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55860 (local address 127.12.12.60:44397)
0621 07:52:11.059775 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.059784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.059825 (+ 41us) server_negotiation.cc:407] Connection header received
0621 07:52:11.060018 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.060022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.060078 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.060650 (+ 572us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.060753 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.061585 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.062958 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.064176 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.064727 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.066918 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.066924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.066930 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.066961 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.070255 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.070776 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.070779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.070782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.070886 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.071183 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.071186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.071188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.071414 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.071502 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.071788 (+ 286us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.071900 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":113,"threads_started":1}
I20260621 07:52:11.072950 18445 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.12.60:44397: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c"
member_type: NON_VOTER
last_known_addr {
host: "127.12.12.60"
port: 44397
}
attrs {
promote: true
}
}
dest_uuid: "bae2da0cc2794cc78af1b0c07edafeaf"
cas_config_opid_index: 5
I20260621 07:52:11.073577 18445 consensus_queue.cc:237] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: NON_VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } }
I20260621 07:52:11.075001 19004 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d8699395d82d49d3a6312d71a6287a1c (127.12.12.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: NON_VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } } }
I20260621 07:52:11.076455 19004 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.12.60:44397
I20260621 07:52:11.076671 19004 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: NON_VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } } }
I20260621 07:52:11.076784 19004 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:11.077328 19009 master.cc:537] Master@127.12.12.60:44397 shutting down...
I20260621 07:52:11.077641 19137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:11.079213 19137 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 1 FOLLOWER]: Refusing update from remote peer bae2da0cc2794cc78af1b0c07edafeaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 07:52:11.079900 19005 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 07:52:11.082000 18430 consensus_peers.cc:487] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf -> Peer d8699395d82d49d3a6312d71a6287a1c (127.12.12.60:44397): Couldn't send request to peer d8699395d82d49d3a6312d71a6287a1c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260621 07:52:11.084421 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.074893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48524 (local address 127.12.12.61:35893)
0621 07:52:11.075030 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.075035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.077561 (+ 2526us) server_negotiation.cc:407] Connection header received
0621 07:52:11.077623 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.077628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.077700 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.077791 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:11.078167 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.078994 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.082204 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.083031 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.083138 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.083301 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.084150 (+ 849us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.084244 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:52:11.084457 19009 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:11.084746 19009 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:11.084805 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.074728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:35893 (local address 127.0.0.1:48524)
0621 07:52:11.077380 (+ 2652us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.077395 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.077478 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.077923 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.077927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.077944 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:11.078032 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.078038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.079123 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.079126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.082055 (+ 2929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.082067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.083828 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.083832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.083928 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.083990 (+ 62us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.084067 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.084695 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2567}
I20260621 07:52:11.085232 19009 tablet_replica.cc:330] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: stopping tablet replica
I20260621 07:52:11.085322 19055 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 LEARNER]: Refusing update from remote peer bae2da0cc2794cc78af1b0c07edafeaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 07:52:11.085770 19005 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: NON_VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260621 07:52:11.087663 19055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d8699395d82d49d3a6312d71a6287a1c (127.12.12.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: NON_VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } } }
I20260621 07:52:11.089929 19084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bae2da0cc2794cc78af1b0c07edafeaf. Latest consensus state: current_term: 1 leader_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: NON_VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } } }
I20260621 07:52:11.090031 19084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: This master's current role is: LEARNER
I20260621 07:52:11.090121 19084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: NON_VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } } }
I20260621 07:52:11.090171 19084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: This master's current role is: LEARNER
I20260621 07:52:11.091702 19005 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf: attempting to promote NON_VOTER 2aeaf63573984aafa402286856776296 to VOTER
I20260621 07:52:11.092150 19005 consensus_queue.cc:237] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } }
I20260621 07:52:11.093398 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.093089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:44397 (local address 127.0.0.1:55862)
0621 07:52:11.093238 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.093306 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260621 07:52:11.093883 19055 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 LEARNER]: Refusing update from remote peer bae2da0cc2794cc78af1b0c07edafeaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260621 07:52:11.094254 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.094041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:44397 (local address 127.0.0.1:55872)
0621 07:52:11.094156 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.094197 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260621 07:52:11.094501 19004 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260621 07:52:11.094877 18430 consensus_peers.cc:487] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf -> Peer d8699395d82d49d3a6312d71a6287a1c (127.12.12.60:44397): Couldn't send request to peer d8699395d82d49d3a6312d71a6287a1c. Status: Network error: Client connection negotiation failed: client connection to 127.12.12.60:44397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260621 07:52:11.096148 19193 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2aeaf63573984aafa402286856776296 (127.12.12.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } } }
I20260621 07:52:11.096570 19055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2aeaf63573984aafa402286856776296 (127.12.12.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } } }
I20260621 07:52:11.098140 19192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } } }
I20260621 07:52:11.098248 19192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:52:11.099967 19004 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } } }
I20260621 07:52:11.100088 19004 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:11.100265 19004 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } } }
I20260621 07:52:11.100342 19004 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:11.101024 19009 master.cc:552] Master@127.12.12.60:44397 shutdown complete.
I20260621 07:52:11.101068 19009 master_runner.cc:305] Clearing existing system tablet
I20260621 07:52:11.101305 19009 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:11.101363 19009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:11.102741 19009 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 07:52:11.103672 19196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:11.104007 19009 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:11.104271 19009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 07:52:11.104363 19009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/wal
uuid: "d8699395d82d49d3a6312d71a6287a1c"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "c102c765caefc73b25bfc12542ac81b2"
server_key_iv: "ba51f9c25dfe5d2b6a9601a8d7f5d2d6"
server_key_version: "encryptionkey@0"
I20260621 07:52:11.106488 19009 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:11.108944 19009 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 07:52:11.109100 19009 log.cc:1196] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260621 07:52:11.109522 19009 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: Deleting consensus metadata
I20260621 07:52:11.109730 19009 master_runner.cc:315] Copying system tablet from 127.12.12.62:39045
I20260621 07:52:11.111099 19009 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: tablet copy: Beginning tablet copy session from remote peer at address 127.12.12.62:39045
I20260621 07:52:11.120695 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.111820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.12.12.62:39045)
0621 07:52:11.111952 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.111957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.112129 (+ 172us) server_negotiation.cc:407] Connection header received
0621 07:52:11.112246 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.112249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.112300 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.112404 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.112893 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.113722 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.114975 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.115319 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.115432 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.117076 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.117080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.117083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.117112 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.118861 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.119372 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.119376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.119378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.119472 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.119769 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.119773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.119775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.120010 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.120112 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.120412 (+ 300us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.120534 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 07:52:11.120703 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.111717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:49298)
0621 07:52:11.112021 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.112035 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.112159 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.112441 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.112444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.112660 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:11.112767 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.112774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.113849 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.113853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.114836 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.114847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.115475 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.115479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.115587 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.116306 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:11.116328 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:11.116901 (+ 573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:11.118983 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.118986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.118988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.119251 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.119571 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.119574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.119576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.119672 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.120113 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:11.120117 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:11.120225 (+ 108us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.120336 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.120493 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":89,"threads_started":1}
I20260621 07:52:11.121559 18465 tablet_copy_service.cc:140] P bae2da0cc2794cc78af1b0c07edafeaf: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d8699395d82d49d3a6312d71a6287a1c ({username='oryx', principal='oryx/127.12.12.60@KRBTEST.COM'} at 127.0.0.1:49298)
I20260621 07:52:11.121654 18465 tablet_copy_service.cc:161] P bae2da0cc2794cc78af1b0c07edafeaf: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d8699395d82d49d3a6312d71a6287a1c at {username='oryx', principal='oryx/127.12.12.60@KRBTEST.COM'} at 127.0.0.1:49298: session id = d8699395d82d49d3a6312d71a6287a1c-00000000000000000000000000000000
I20260621 07:52:11.122949 18465 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 07:52:11.124373 19009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 07:52:11.127027 19009 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: tablet copy: Starting download of 0 data blocks...
I20260621 07:52:11.127308 19009 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: tablet copy: Starting download of 1 WAL segments...
I20260621 07:52:11.128978 19009 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 07:52:11.130951 18465 tablet_copy_service.cc:342] P bae2da0cc2794cc78af1b0c07edafeaf: Request end of tablet copy session d8699395d82d49d3a6312d71a6287a1c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.12.60@KRBTEST.COM'} at 127.0.0.1:49298
I20260621 07:52:11.131052 18465 tablet_copy_service.cc:433] P bae2da0cc2794cc78af1b0c07edafeaf: ending tablet copy session d8699395d82d49d3a6312d71a6287a1c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d8699395d82d49d3a6312d71a6287a1c
I20260621 07:52:11.138334 19009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:11.140262 19203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:11.140290 19206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:11.140640 19204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:11.140756 19009 server_base.cc:1034] running on GCE node
I20260621 07:52:11.141018 19009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:11.141305 19009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:11.142514 19009 hybrid_clock.cc:648] HybridClock initialized: now 1782028331142499 us; error 33 us; skew 500 ppm
Jun 21 07:52:11 dist-test-slave-lbpt krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028331, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:11.144114 19009 init.cc:377] Logged in from keytab as oryx/127.12.12.60@KRBTEST.COM (short username oryx)
I20260621 07:52:11.144598 19009 webserver.cc:460] Webserver started at http://127.12.12.60:37615/ using document root <none> and password file <none>
I20260621 07:52:11.144795 19009 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:11.144837 19009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:11.146129 19009 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 07:52:11.147221 19213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:11.147514 19009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 07:52:11.147593 19009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/wal
uuid: "d8699395d82d49d3a6312d71a6287a1c"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "c102c765caefc73b25bfc12542ac81b2"
server_key_iv: "ba51f9c25dfe5d2b6a9601a8d7f5d2d6"
server_key_version: "encryptionkey@0"
I20260621 07:52:11.147675 19009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:11.178613 19009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:11.181159 19009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:11.186430 19009 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.60:44397
I20260621 07:52:11.186470 19265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.60:44397 every 8 connection(s)
I20260621 07:52:11.187036 19009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/master-2/data/info.pb
I20260621 07:52:11.188863 19266 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:52:11.189973 19266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: Bootstrap starting.
I20260621 07:52:11.201582 19266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:52:11.201913 19266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: Bootstrap complete.
I20260621 07:52:11.202569 19266 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } }
I20260621 07:52:11.202714 19266 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8699395d82d49d3a6312d71a6287a1c, State: Initialized, Role: LEARNER
I20260621 07:52:11.202821 19266 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } }
I20260621 07:52:11.203184 19269 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } } }
I20260621 07:52:11.203281 19266 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:52:11.203415 19269 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [sys.catalog]: This master's current role is: LEARNER
I20260621 07:52:11.208398 19283 catalog_manager.cc:1259] Loaded cluster ID: d3f80f6fa24041078a598910c09aaaf3
I20260621 07:52:11.208472 19283 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: loading cluster ID for follower catalog manager: success
I20260621 07:52:11.208606 18770 heartbeater.cc:503] Master 127.12.12.62:39045 was elected leader, sending a full tablet report...
I20260621 07:52:11.210858 19283 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: acquiring CA information for follower catalog manager: success
W20260621 07:52:11.211268 19189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:11.211558 19189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:11.211642 19189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260621 07:52:11.212395 19283 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 07:52:11.215977 19189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:11.216059 19189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:11.216176 19189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:52:11.221643 19189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.1.keytab
--principal=oryx/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:44609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=44647
--tserver_master_addrs=127.12.12.62:39045,127.12.12.61:35893,127.12.12.60:44397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:11.222925 19189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:11.224102 19189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:11.233955 19285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:11.233955 19288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:11.233999 19286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:11.234297 19189 server_base.cc:1034] running on GCE node
I20260621 07:52:11.234747 19189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:11.235509 19189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:11.236711 19189 hybrid_clock.cc:648] HybridClock initialized: now 1782028331236696 us; error 30 us; skew 500 ppm
Jun 21 07:52:11 dist-test-slave-lbpt krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028331, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:11.240459 19189 init.cc:377] Logged in from keytab as oryx/127.12.12.1@KRBTEST.COM (short username oryx)
I20260621 07:52:11.241493 19189 webserver.cc:460] Webserver started at http://127.12.12.1:44647/ using document root <none> and password file <none>
I20260621 07:52:11.242164 19189 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:11.242246 19189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:11.247712 19189 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:52:11.250064 19295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:11.251407 19189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:11.251540 19189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "85929ce3cf294504a55fd8e9ad3515ba"
format_stamp: "Formatted at 2026-06-21 07:52:09 on dist-test-slave-lbpt"
server_key: "e032c55b3c7276d878cfb0bc458cdea1"
server_key_iv: "da8d3226a715bfd713ddf7f6dec6e811"
server_key_version: "encryptionkey@0"
I20260621 07:52:11.252107 19189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:11.271941 19189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:11.275557 19189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:11.275794 19189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:11.276425 19189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:11.277318 19189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:11.277387 19189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:11.277460 19189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:11.277504 19189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:11.287153 19189 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:44609
I20260621 07:52:11.287184 19408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:44609 every 8 connection(s)
I20260621 07:52:11.288218 19189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:52:11.296116 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 19189
I20260621 07:52:11.296329 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 18641
Jun 21 07:52:11 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028331, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.61@KRBTEST.COM
Jun 21 07:52:11 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028331, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.60@KRBTEST.COM
Jun 21 07:52:11 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028331, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM
I20260621 07:52:11.305876 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.290899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:42065 (local address 127.12.12.61:35893)
0621 07:52:11.291061 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.291065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.291661 (+ 596us) server_negotiation.cc:407] Connection header received
0621 07:52:11.292851 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.292854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.292907 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.293031 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.294599 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.295151 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.296863 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.297275 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.297372 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.301082 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.301089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.301096 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.301132 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.302967 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.303684 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.303691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.303696 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.303789 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.304263 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.304269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.304274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.304546 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.304658 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.305539 (+ 881us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.305671 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 07:52:11.305857 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:42563
--local_ip_for_outbound_sockets=127.12.12.2
--tserver_master_addrs=127.12.12.62:39045
--webserver_port=37421
--webserver_interface=127.12.12.2
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.2.keytab
--principal=oryx/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.12.62:39045,127.12.12.61:35893
--tserver_master_addrs=127.12.12.62:39045,127.12.12.61:35893,127.12.12.60:44397 with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:52:11.306288 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.290171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:32915 (local address 127.12.12.62:39045)
0621 07:52:11.290340 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.290345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.291451 (+ 1106us) server_negotiation.cc:407] Connection header received
0621 07:52:11.293016 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.293020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.293071 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.293145 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.294670 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.295189 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.297105 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.297480 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.297555 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.301951 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.301956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.301960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.301996 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.303943 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.304666 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.304670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.304672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.304773 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.305094 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.305098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.305100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.305334 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.305422 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.305814 (+ 392us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.306146 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 07:52:11.306906 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.291923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:56755 (local address 127.12.12.60:44397)
0621 07:52:11.292215 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.292220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.293187 (+ 967us) server_negotiation.cc:407] Connection header received
0621 07:52:11.293312 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.293316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.293368 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.293486 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.295320 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.296410 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.297534 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.297886 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.297986 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.301297 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.301301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.301304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.301333 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.304021 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.304666 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.304670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.304672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.304773 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.305094 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.305098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.305100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.305340 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.305423 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.306622 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.306752 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":81,"threads_started":1}
I20260621 07:52:11.307734 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.290683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.1:32915)
0621 07:52:11.291305 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.291356 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.292890 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.293272 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.293329 (+ 57us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.293871 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:11.294458 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.294470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.295339 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.295342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.296974 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.296983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.297703 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.297707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.297933 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.298952 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:11.298968 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:11.301752 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:11.304138 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.304141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.304143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.304547 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.304886 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.304888 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.304891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.304994 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.305472 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:11.305474 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:11.305607 (+ 133us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.306556 (+ 949us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.306699 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":489,"spinlock_wait_cycles":12416,"thread_start_us":160,"threads_started":1}
I20260621 07:52:11.307734 19414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.290720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:35893 (local address 127.12.12.1:42065)
0621 07:52:11.291582 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.291594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.292612 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.293269 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.293280 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.293902 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:11.294436 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.294456 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.295302 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.295306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.296716 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.296729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.297632 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.297638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.297936 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.298709 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:11.298735 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:11.300872 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:11.303103 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.303119 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.303140 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.303543 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.303953 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.303959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.303964 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.304155 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.304805 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:11.304815 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:11.305222 (+ 407us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.306290 (+ 1068us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.307164 (+ 874us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":270,"spinlock_wait_cycles":53888,"thread_start_us":154,"threads_started":1}
I20260621 07:52:11.308034 19415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.290890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:44397 (local address 127.12.12.1:56755)
0621 07:52:11.293107 (+ 2217us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.293120 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.293221 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.293503 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.293506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.293902 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:11.294436 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.294456 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.296551 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.296556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.297405 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.297414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.298118 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.298124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.298269 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.298861 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:11.298880 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:11.301113 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:11.304138 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.304141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.304143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.304547 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.304881 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.304885 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.304886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.304988 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.305469 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:11.305472 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:11.305610 (+ 138us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.305736 (+ 126us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.306150 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2155,"mutex_wait_us":86,"spinlock_wait_cycles":50176,"thread_start_us":640,"threads_started":1}
I20260621 07:52:11.309355 19409 heartbeater.cc:347] Connected to a master server at 127.12.12.60:44397
I20260621 07:52:11.309393 19411 heartbeater.cc:347] Connected to a master server at 127.12.12.61:35893
I20260621 07:52:11.309501 19410 heartbeater.cc:347] Connected to a master server at 127.12.12.62:39045
I20260621 07:52:11.309633 19410 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:11.309629 19409 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:11.309629 19411 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:11.310313 19410 heartbeater.cc:511] Master 127.12.12.62:39045 requested a full tablet report, sending...
I20260621 07:52:11.310348 19409 heartbeater.cc:511] Master 127.12.12.60:44397 requested a full tablet report, sending...
I20260621 07:52:11.310312 19411 heartbeater.cc:511] Master 127.12.12.61:35893 requested a full tablet report, sending...
I20260621 07:52:11.311743 18445 ts_manager.cc:194] Re-registered known tserver with Master: 85929ce3cf294504a55fd8e9ad3515ba (127.12.12.1:44609)
I20260621 07:52:11.312201 19228 ts_manager.cc:194] Registered new tserver with Master: 85929ce3cf294504a55fd8e9ad3515ba (127.12.12.1:44609)
I20260621 07:52:11.312408 18445 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:32915
I20260621 07:52:11.313491 19045 ts_manager.cc:194] Registered new tserver with Master: 85929ce3cf294504a55fd8e9ad3515ba (127.12.12.1:44609)
W20260621 07:52:11.425827 19417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:11.426127 19417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:11.426170 19417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:11.430575 19417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:11.430634 19417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:11.430732 19417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:52:11.436471 19417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.2.keytab
--principal=oryx/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:42563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=37421
--tserver_master_addrs=127.12.12.62:39045,127.12.12.61:35893,127.12.12.60:44397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:11.437573 19417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:11.438081 18904 heartbeater.cc:503] Master 127.12.12.62:39045 was elected leader, sending a full tablet report...
I20260621 07:52:11.438875 19417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:11.448918 19422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:11.448908 19423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:11.448946 19425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:11.449586 19417 server_base.cc:1034] running on GCE node
I20260621 07:52:11.450029 19417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:11.450794 19417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:11.451998 19417 hybrid_clock.cc:648] HybridClock initialized: now 1782028331451965 us; error 55 us; skew 500 ppm
Jun 21 07:52:11 dist-test-slave-lbpt krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028331, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:11.454999 19417 init.cc:377] Logged in from keytab as oryx/127.12.12.2@KRBTEST.COM (short username oryx)
I20260621 07:52:11.456034 19417 webserver.cc:460] Webserver started at http://127.12.12.2:37421/ using document root <none> and password file <none>
I20260621 07:52:11.456719 19417 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:11.456833 19417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:11.460813 19417 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:52:11.463057 19432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:11.464408 19417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:11.464556 19417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "1de5e565c961429ab12b8c5e25f72c9f"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "f440182bff9d57962db50fee6c9bbf93"
server_key_iv: "0546e6073d96682cc642e6feb6576e69"
server_key_version: "encryptionkey@0"
I20260621 07:52:11.465145 19417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:11.507221 19417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:11.510865 19417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:11.511080 19417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:11.511729 19417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:11.512641 19417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:11.512688 19417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:11.512759 19417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:11.512797 19417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:11.521823 19417 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:42563
I20260621 07:52:11.521842 19545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:42563 every 8 connection(s)
I20260621 07:52:11.522918 19417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:52:11.524389 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 19417
I20260621 07:52:11.524564 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 18775
I20260621 07:52:11.532418 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:42983
--local_ip_for_outbound_sockets=127.12.12.3
--tserver_master_addrs=127.12.12.62:39045
--webserver_port=46713
--webserver_interface=127.12.12.3
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.3.keytab
--principal=oryx/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.12.62:39045,127.12.12.61:35893
--tserver_master_addrs=127.12.12.62:39045,127.12.12.61:35893,127.12.12.60:44397 with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:52:11 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028331, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.60@KRBTEST.COM
Jun 21 07:52:11 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028331, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.61@KRBTEST.COM
Jun 21 07:52:11 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028331, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM
I20260621 07:52:11.542642 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.525738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:55049 (local address 127.12.12.61:35893)
0621 07:52:11.526368 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.526372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.527881 (+ 1509us) server_negotiation.cc:407] Connection header received
0621 07:52:11.528395 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.528401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.528465 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.528515 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.530626 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.531496 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.533480 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.533810 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.533887 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.537938 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.537942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.537944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.537974 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.539927 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.540705 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.540711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.540715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.540796 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.541216 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.541219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.541221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.541472 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.541551 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.542309 (+ 758us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.542470 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564}
I20260621 07:52:11.542667 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.525171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:44145 (local address 127.12.12.62:39045)
0621 07:52:11.525315 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.525319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.527508 (+ 2189us) server_negotiation.cc:407] Connection header received
0621 07:52:11.528293 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.528300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.528369 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.528424 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.530626 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.531460 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.533463 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.533816 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.533887 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.538595 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.538599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.538602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.538635 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.540543 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.541032 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.541035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.541037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.541113 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.541400 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.541403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.541405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.541616 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.541686 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.542300 (+ 614us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.542534 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:52:11.542974 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.525379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:56275 (local address 127.12.12.60:44397)
0621 07:52:11.526284 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.526288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.526847 (+ 559us) server_negotiation.cc:407] Connection header received
0621 07:52:11.528289 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.528296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.528366 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.528437 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.530622 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.531497 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.533471 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.533882 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.533981 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.537825 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.537832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.537836 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.537874 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.540777 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.541339 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.541344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.541348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.541453 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.541806 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.541811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.541815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.542066 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.542162 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.542596 (+ 434us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.542748 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818}
I20260621 07:52:11.544448 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.525473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.2:44145)
0621 07:52:11.527406 (+ 1933us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.527423 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.528042 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.528861 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.528871 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.529570 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:11.530440 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.530462 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.531656 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.531659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.533314 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.533325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.534101 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.534105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.534503 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.535407 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:11.535419 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:11.538377 (+ 2958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:11.540670 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.540676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.540681 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.540927 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.541218 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.541221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.541222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.541300 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.541735 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:11.541738 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:11.542003 (+ 265us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.542889 (+ 886us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.543259 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1859,"mutex_wait_us":617,"spinlock_wait_cycles":62336,"thread_start_us":121,"threads_started":1}
I20260621 07:52:11.544450 19550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.525236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:44397 (local address 127.12.12.2:56275)
0621 07:52:11.526668 (+ 1432us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.526725 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.528037 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.528862 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.528872 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.529573 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:11.530465 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.530479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.531657 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.531660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.533314 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.533328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.534178 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.534184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.534490 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.535571 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:11.535591 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:11.537606 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:11.540930 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.540935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.540940 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.541213 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.541581 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.541586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.541591 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.541690 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.542228 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:11.542234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:11.542343 (+ 109us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.543193 (+ 850us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.543370 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1199,"mutex_wait_us":394,"spinlock_wait_cycles":3968,"thread_start_us":693,"threads_started":1}
I20260621 07:52:11.545351 19551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.527153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:35893 (local address 127.12.12.2:55049)
0621 07:52:11.527785 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.527796 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.528266 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.528861 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.528871 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.529570 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:11.530440 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.530462 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.531656 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.531659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.533353 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.533364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.534101 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.534105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.534515 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.535269 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:11.535373 (+ 104us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:11.537762 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:11.540218 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.540225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.540245 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.540596 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.540913 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.540916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.540919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.541093 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.541615 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:11.541622 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:11.542003 (+ 381us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.543461 (+ 1458us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.543591 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":688,"spinlock_wait_cycles":86784,"thread_start_us":116,"threads_started":1}
I20260621 07:52:11.545840 19546 heartbeater.cc:347] Connected to a master server at 127.12.12.60:44397
I20260621 07:52:11.545924 19547 heartbeater.cc:347] Connected to a master server at 127.12.12.62:39045
I20260621 07:52:11.545962 19548 heartbeater.cc:347] Connected to a master server at 127.12.12.61:35893
I20260621 07:52:11.546084 19548 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:11.546085 19546 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:11.546084 19547 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:11.546772 19547 heartbeater.cc:511] Master 127.12.12.62:39045 requested a full tablet report, sending...
I20260621 07:52:11.546836 19548 heartbeater.cc:511] Master 127.12.12.61:35893 requested a full tablet report, sending...
I20260621 07:52:11.546774 19546 heartbeater.cc:511] Master 127.12.12.60:44397 requested a full tablet report, sending...
I20260621 07:52:11.547971 19228 ts_manager.cc:194] Registered new tserver with Master: 1de5e565c961429ab12b8c5e25f72c9f (127.12.12.2:42563)
I20260621 07:52:11.547991 19045 ts_manager.cc:194] Registered new tserver with Master: 1de5e565c961429ab12b8c5e25f72c9f (127.12.12.2:42563)
I20260621 07:52:11.548094 18445 ts_manager.cc:194] Re-registered known tserver with Master: 1de5e565c961429ab12b8c5e25f72c9f (127.12.12.2:42563)
I20260621 07:52:11.548866 18445 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:44145
I20260621 07:52:11.594628 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.590681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:44397 (local address 127.0.0.1:55876)
0621 07:52:11.590830 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.590848 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.590949 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.591331 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.591334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.591355 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:11.591459 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.591465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.592493 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.592496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.593260 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.593270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.594176 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.594180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.594267 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.594289 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.594371 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.594513 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260621 07:52:11.594630 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.590772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.12.12.60:44397)
0621 07:52:11.590909 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.590914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.590934 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:52:11.591073 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.591076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.591156 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.591282 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:11.591582 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.592350 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.593396 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.594032 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.594140 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.594302 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.594389 (+ 87us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.594493 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 07:52:11.595424 19240 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260621 07:52:11.595868 19269 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bae2da0cc2794cc78af1b0c07edafeaf. Latest consensus state: current_term: 1 leader_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } } }
I20260621 07:52:11.596042 19269 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [sys.catalog]: This master's current role is: LEARNER
W20260621 07:52:11.655516 19553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:11.655802 19553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:11.655848 19553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:11.660210 19553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:11.660281 19553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:11.660344 19553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:52:11.665944 19553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.3.keytab
--principal=oryx/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:42983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=46713
--tserver_master_addrs=127.12.12.62:39045,127.12.12.61:35893,127.12.12.60:44397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:11.667291 19553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:11.668481 19553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:11.678841 19559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:11.678861 19553 server_base.cc:1034] running on GCE node
W20260621 07:52:11.678781 19558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:11.678800 19561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:11.679489 19553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:11.680125 19553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:11.681283 19553 hybrid_clock.cc:648] HybridClock initialized: now 1782028331681264 us; error 35 us; skew 500 ppm
Jun 21 07:52:11 dist-test-slave-lbpt krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028331, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:11.684294 19553 init.cc:377] Logged in from keytab as oryx/127.12.12.3@KRBTEST.COM (short username oryx)
I20260621 07:52:11.685305 19553 webserver.cc:460] Webserver started at http://127.12.12.3:46713/ using document root <none> and password file <none>
I20260621 07:52:11.685876 19553 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:11.685966 19553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:11.689754 19553 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:52:11.691814 19568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:11.693065 19553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:52:11.693187 19553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "5906911d01134202bed9be4d170b2875"
format_stamp: "Formatted at 2026-06-21 07:52:10 on dist-test-slave-lbpt"
server_key: "8442aeb7740b735c7487275790bd6e16"
server_key_iv: "7b00b653f686a315c4859a6080c65ace"
server_key_version: "encryptionkey@0"
I20260621 07:52:11.693732 19553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:11.715441 19553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:11.718966 19553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:11.719184 19553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:11.719827 19553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:11.720712 19553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:11.720758 19553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:11.720798 19553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:11.720851 19553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:11.730475 19553 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:42983
I20260621 07:52:11.730576 19681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:42983 every 8 connection(s)
I20260621 07:52:11.731608 19553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:52:11.740365 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 19553
Jun 21 07:52:11 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028331, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM
Jun 21 07:52:11 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028331, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.61@KRBTEST.COM
Jun 21 07:52:11 dist-test-slave-lbpt krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028331, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.60@KRBTEST.COM
I20260621 07:52:11.749387 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.733695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:35513 (local address 127.12.12.62:39045)
0621 07:52:11.733859 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.733865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.735520 (+ 1655us) server_negotiation.cc:407] Connection header received
0621 07:52:11.736305 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.736309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.736369 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.736426 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.737919 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.738765 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.740139 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.740491 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.740617 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.744943 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.744947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.744949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.744972 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.746808 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.747449 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.747453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.747454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.747539 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.747977 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.747983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.747988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.748230 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.748314 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.749070 (+ 756us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.749196 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:52:11.751317 19687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.734269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.3:35513)
0621 07:52:11.735399 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.735414 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.736045 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.736637 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.736647 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.737156 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:11.737769 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.737784 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.738882 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.738885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.739988 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.739995 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.740650 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.740654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.740985 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.741778 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:11.741805 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:11.744742 (+ 2937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:11.746957 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.746964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.746983 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.747324 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.747654 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.747658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.747660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.747842 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.748403 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:11.748410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:11.748751 (+ 341us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.749562 (+ 811us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.749938 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"mutex_wait_us":291,"spinlock_wait_cycles":20992,"thread_start_us":130,"threads_started":1}
I20260621 07:52:11.751695 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.734247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:59395 (local address 127.12.12.60:44397)
0621 07:52:11.734380 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.734385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.735004 (+ 619us) server_negotiation.cc:407] Connection header received
0621 07:52:11.736356 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.736359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.736422 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.736474 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.737920 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.738765 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.742507 (+ 3742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.742841 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.742952 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.745338 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.745342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.745344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.745373 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.749111 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.749765 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.749769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.749770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.750107 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.750476 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.750480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.750481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.750838 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.750917 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.751413 (+ 496us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.751554 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 07:52:11.751946 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.734269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:44397 (local address 127.12.12.3:59395)
0621 07:52:11.734837 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.734893 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.736045 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.736637 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.736647 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.737156 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:11.737770 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.737777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.738951 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.738953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.739872 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.739882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.743100 (+ 3218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.743109 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.743210 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.743709 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:11.743725 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:11.745184 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:11.749225 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.749228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.749230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.749491 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.750214 (+ 723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.750217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.750218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.750343 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.750970 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:11.750973 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:11.751059 (+ 86us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.751152 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.751277 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":154,"spinlock_wait_cycles":42496,"thread_start_us":162,"threads_started":1}
I20260621 07:52:11.752940 19682 heartbeater.cc:347] Connected to a master server at 127.12.12.60:44397
I20260621 07:52:11.753182 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.742300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49304 (local address 127.12.12.62:39045)
0621 07:52:11.742693 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.742698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.742939 (+ 241us) server_negotiation.cc:407] Connection header received
0621 07:52:11.743042 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.743045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.743096 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.743196 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.744823 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.745777 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.747268 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.747694 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.747776 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.749539 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.749544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.749550 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.749575 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.751322 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.751810 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.751813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.751815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.751902 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.752238 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.752241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.752243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.752449 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.752552 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.752912 (+ 360us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.753038 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":79,"threads_started":1}
I20260621 07:52:11.753197 19682 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:11.753728 19688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.734269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:35893 (local address 127.12.12.3:53365)
0621 07:52:11.735646 (+ 1377us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.735659 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.736057 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.736637 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.736645 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.737147 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:11.737755 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.737769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.738669 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.738672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.739980 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.739990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.740650 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.740654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.740985 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.741869 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:11.741884 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:11.744742 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:11.752090 (+ 7348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.752096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.752101 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.752343 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.752679 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:11.752682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:11.752684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:11.752765 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:11.753168 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:11.753172 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:11.753275 (+ 103us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.753367 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.753529 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1295,"mutex_wait_us":411,"spinlock_wait_cycles":31360,"thread_start_us":118,"threads_started":1}
I20260621 07:52:11.753782 19682 heartbeater.cc:511] Master 127.12.12.60:44397 requested a full tablet report, sending...
I20260621 07:52:11.754241 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.733692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:53365 (local address 127.12.12.61:35893)
0621 07:52:11.733844 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.733849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.735726 (+ 1877us) server_negotiation.cc:407] Connection header received
0621 07:52:11.736287 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.736291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.736351 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.736436 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.737902 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.738510 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.740122 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.740379 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.740448 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.744936 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.744941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.744943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.744971 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.747455 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.752479 (+ 5024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.752483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.752485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.752581 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.752858 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.752861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.752863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.753059 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.753168 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.753963 (+ 795us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.754090 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 07:52:11.754609 19683 heartbeater.cc:347] Connected to a master server at 127.12.12.62:39045
I20260621 07:52:11.754688 19683 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:11.754848 19683 heartbeater.cc:511] Master 127.12.12.62:39045 requested a full tablet report, sending...
I20260621 07:52:11.754912 19684 heartbeater.cc:347] Connected to a master server at 127.12.12.61:35893
I20260621 07:52:11.755003 19684 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:11.755052 19228 ts_manager.cc:194] Registered new tserver with Master: 5906911d01134202bed9be4d170b2875 (127.12.12.3:42983)
I20260621 07:52:11.755179 19684 heartbeater.cc:511] Master 127.12.12.61:35893 requested a full tablet report, sending...
I20260621 07:52:11.755697 18444 ts_manager.cc:194] Re-registered known tserver with Master: 5906911d01134202bed9be4d170b2875 (127.12.12.3:42983)
I20260621 07:52:11.755896 19045 ts_manager.cc:194] Registered new tserver with Master: 5906911d01134202bed9be4d170b2875 (127.12.12.3:42983)
I20260621 07:52:11.756402 18444 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:35513
I20260621 07:52:11.761225 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.746145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.12.12.60:44397)
0621 07:52:11.746588 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.746592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.747136 (+ 544us) server_negotiation.cc:407] Connection header received
0621 07:52:11.747305 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.747308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.747355 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.747413 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.748363 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.749328 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.751035 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.751682 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.755146 (+ 3464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.757246 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.757250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.757253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.757279 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.759184 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.759690 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.759694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.759696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.759781 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.760073 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.760077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.760079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.760297 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.760419 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.760747 (+ 328us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.761037 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":97,"threads_started":1}
I20260621 07:52:11.759415 18445 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 07:52:11.763631 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.743740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.12.12.61:35893)
0621 07:52:11.753282 (+ 9542us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.753287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.753299 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:52:11.753332 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.753338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.753395 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.754330 (+ 935us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.754456 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.756498 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.757509 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.758309 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.758771 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.759965 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.759969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.759972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.760000 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.761863 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.762314 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.762316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.762318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.762392 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.762756 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.762759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.762761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.762974 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.763091 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.763323 (+ 232us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.763443 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
W20260621 07:52:11.763973 18445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:52:11.766901 19700 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: NON_VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: true } } }
I20260621 07:52:11.767071 19700 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:11.770125 19700 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf: attempting to promote NON_VOTER d8699395d82d49d3a6312d71a6287a1c to VOTER
I20260621 07:52:11.770547 19700 consensus_queue.cc:237] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } }
I20260621 07:52:11.771626 19240 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 1 LEARNER]: Refusing update from remote peer bae2da0cc2794cc78af1b0c07edafeaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260621 07:52:11.772048 19193 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260621 07:52:11.772110 19055 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 FOLLOWER]: Refusing update from remote peer bae2da0cc2794cc78af1b0c07edafeaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260621 07:52:11.773116 19193 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260621 07:52:11.775084 19701 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d8699395d82d49d3a6312d71a6287a1c (127.12.12.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } } }
I20260621 07:52:11.775307 19240 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d8699395d82d49d3a6312d71a6287a1c (127.12.12.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } } }
I20260621 07:52:11.776594 19193 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } } }
I20260621 07:52:11.776738 19193 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:11.776765 19701 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } } }
I20260621 07:52:11.776851 19701 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:11.776952 19269 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } } }
I20260621 07:52:11.777081 19269 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:52:11.776866 19193 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } } }
I20260621 07:52:11.777201 19193 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:11.777472 19055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d8699395d82d49d3a6312d71a6287a1c (127.12.12.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } } }
I20260621 07:52:11.780160 19710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } } }
I20260621 07:52:11.780298 19710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:52:11.787437 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.782720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.12.12.2:42563)
0621 07:52:11.783042 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.783052 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.783070 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:52:11.783131 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.783136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.783274 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.783416 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:11.783814 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.784919 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.786161 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.786843 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.786967 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.787127 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.787184 (+ 57us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.787275 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":106,"threads_started":1}
I20260621 07:52:11.787441 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.782266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:42563 (local address 127.0.0.1:41140)
0621 07:52:11.782585 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.782599 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.782680 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.783594 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.783600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.783620 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:11.783693 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.783698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.785036 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.785039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.786034 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.786043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.786972 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.786976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.787062 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.787090 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.787159 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.787234 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":21,"thread_start_us":106,"threads_started":1}
I20260621 07:52:11.787915 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.781894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:42983 (local address 127.0.0.1:33902)
0621 07:52:11.782207 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.782222 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.782312 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.783307 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.783310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.783329 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:11.783456 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.783465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.784707 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.784710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.786172 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.786183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.787477 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.787483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.787598 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.787630 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.787686 (+ 56us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.787760 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":4,"thread_start_us":135,"threads_started":1}
I20260621 07:52:11.788209 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.781563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:44609 (local address 127.0.0.1:37102)
0621 07:52:11.781713 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.781729 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.781812 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.783416 (+ 1604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.783419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.783436 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:11.783523 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.783528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.785504 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.785508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.786720 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.786730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.787754 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.787758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.787866 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.787887 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.787979 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.788054 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260621 07:52:11.788228 19712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.782004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.12.12.1:44609)
0621 07:52:11.782916 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.782932 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.782955 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:52:11.783033 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.783039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.783240 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.783664 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:11.783814 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.785388 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.786861 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.787615 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.787754 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.787896 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.788006 (+ 110us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.788092 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":80,"threads_started":1}
I20260621 07:52:11.789316 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.782353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33902 (local address 127.12.12.3:42983)
0621 07:52:11.782704 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.782718 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.782742 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:52:11.782851 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.782858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.783123 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.783341 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:11.783624 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.784541 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.786340 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.787333 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.788961 (+ 1628us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.789084 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.789128 (+ 44us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.789202 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":127,"threads_started":1}
I20260621 07:52:11.791020 19343 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3c092b9249484eaac62362586e6f3e (DEFAULT_TABLE table=test-table [id=c0f3e6c484784e60bf587781df050fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:11.791019 19480 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3c092b9249484eaac62362586e6f3e (DEFAULT_TABLE table=test-table [id=c0f3e6c484784e60bf587781df050fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:11.792171 19343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3c092b9249484eaac62362586e6f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:11.792171 19480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3c092b9249484eaac62362586e6f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:11.796998 19616 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3c092b9249484eaac62362586e6f3e (DEFAULT_TABLE table=test-table [id=c0f3e6c484784e60bf587781df050fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:11.798161 19616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3c092b9249484eaac62362586e6f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:11.800685 19716 tablet_bootstrap.cc:492] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba: Bootstrap starting.
I20260621 07:52:11.801128 19717 tablet_bootstrap.cc:492] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f: Bootstrap starting.
I20260621 07:52:11.803658 19717 tablet_bootstrap.cc:654] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:11.804596 19717 log.cc:824] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:11.804623 19716 tablet_bootstrap.cc:654] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:11.805452 19716 log.cc:824] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:11.806895 19720 tablet_bootstrap.cc:492] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875: Bootstrap starting.
I20260621 07:52:11.807633 19716 tablet_bootstrap.cc:492] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba: No bootstrap required, opened a new log
I20260621 07:52:11.807888 19716 ts_tablet_manager.cc:1397] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260621 07:52:11.807947 19717 tablet_bootstrap.cc:492] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f: No bootstrap required, opened a new log
I20260621 07:52:11.808120 19717 ts_tablet_manager.cc:1397] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260621 07:52:11.809175 19720 tablet_bootstrap.cc:654] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:11.810078 19720 log.cc:824] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:11.811520 19716 raft_consensus.cc:348] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } }
I20260621 07:52:11.811760 19716 raft_consensus.cc:374] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:11.811838 19716 raft_consensus.cc:729] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85929ce3cf294504a55fd8e9ad3515ba, State: Initialized, Role: FOLLOWER
I20260621 07:52:11.812239 19716 consensus_queue.cc:260] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } }
I20260621 07:52:11.812865 19410 heartbeater.cc:503] Master 127.12.12.62:39045 was elected leader, sending a full tablet report...
I20260621 07:52:11.812779 19717 raft_consensus.cc:348] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } }
I20260621 07:52:11.813015 19717 raft_consensus.cc:374] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:11.813058 19717 raft_consensus.cc:729] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1de5e565c961429ab12b8c5e25f72c9f, State: Initialized, Role: FOLLOWER
I20260621 07:52:11.813212 19716 ts_tablet_manager.cc:1428] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:52:11.813717 19717 consensus_queue.cc:260] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } }
I20260621 07:52:11.814703 19717 ts_tablet_manager.cc:1428] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260621 07:52:11.815905 19547 heartbeater.cc:503] Master 127.12.12.62:39045 was elected leader, sending a full tablet report...
I20260621 07:52:11.817852 19720 tablet_bootstrap.cc:492] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875: No bootstrap required, opened a new log
I20260621 07:52:11.818056 19720 ts_tablet_manager.cc:1397] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20260621 07:52:11.821481 19720 raft_consensus.cc:348] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } }
I20260621 07:52:11.821683 19720 raft_consensus.cc:374] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:11.821715 19720 raft_consensus.cc:729] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5906911d01134202bed9be4d170b2875, State: Initialized, Role: FOLLOWER
I20260621 07:52:11.822199 19720 consensus_queue.cc:260] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } }
I20260621 07:52:11.823249 19720 ts_tablet_manager.cc:1428] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260621 07:52:11.823534 19683 heartbeater.cc:503] Master 127.12.12.62:39045 was elected leader, sending a full tablet report...
I20260621 07:52:11.844661 19722 raft_consensus.cc:482] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:52:11.844925 19722 raft_consensus.cc:504] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } }
I20260621 07:52:11.846163 19722 leader_election.cc:288] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5906911d01134202bed9be4d170b2875 (127.12.12.3:42983), 1de5e565c961429ab12b8c5e25f72c9f (127.12.12.2:42563)
I20260621 07:52:11.852062 19725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.846496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:42983 (local address 127.12.12.1:35827)
0621 07:52:11.846895 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.846907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.847047 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.847459 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.847462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.847477 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:11.847692 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.847699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.848769 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.848773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.850573 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.850586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.851597 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.851601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.851705 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.851733 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.851813 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.851890 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":40,"thread_start_us":90,"threads_started":1}
I20260621 07:52:11.852097 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.846626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:35827 (local address 127.12.12.3:42983)
0621 07:52:11.846759 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.846767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.846995 (+ 228us) server_negotiation.cc:407] Connection header received
0621 07:52:11.847188 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.847192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.847252 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.847303 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:11.847826 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.848642 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.850729 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.851459 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.851573 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.851649 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.851857 (+ 208us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.851959 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:52:11.852803 19726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.846864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:42563 (local address 127.12.12.1:39105)
0621 07:52:11.847122 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:11.847137 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:11.847206 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:11.847561 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:11.847564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:11.847580 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:11.847708 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.847713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.849540 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.849543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.850858 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:11.850868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.852019 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.852024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:11.852408 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.852437 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 07:52:11.852553 (+ 116us) client_negotiation.cc:240] Negotiation successful
0621 07:52:11.852638 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":24,"thread_start_us":72,"threads_started":1}
I20260621 07:52:11.852783 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.847197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:39105 (local address 127.12.12.2:42563)
0621 07:52:11.847332 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.847337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.847352 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:52:11.847409 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.847412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.847463 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.847541 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:11.848581 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.849377 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.851015 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.851728 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.851848 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.851912 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.852576 (+ 664us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.852666 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:52:11.853063 19636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3c092b9249484eaac62362586e6f3e" candidate_uuid: "85929ce3cf294504a55fd8e9ad3515ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5906911d01134202bed9be4d170b2875" is_pre_election: true
I20260621 07:52:11.853377 19636 raft_consensus.cc:2454] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85929ce3cf294504a55fd8e9ad3515ba in term 0.
I20260621 07:52:11.853439 19500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3c092b9249484eaac62362586e6f3e" candidate_uuid: "85929ce3cf294504a55fd8e9ad3515ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1de5e565c961429ab12b8c5e25f72c9f" is_pre_election: true
I20260621 07:52:11.853814 19500 raft_consensus.cc:2454] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85929ce3cf294504a55fd8e9ad3515ba in term 0.
I20260621 07:52:11.853909 19296 leader_election.cc:302] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5906911d01134202bed9be4d170b2875, 85929ce3cf294504a55fd8e9ad3515ba; no voters:
I20260621 07:52:11.854228 19722 raft_consensus.cc:2781] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:52:11.854362 19722 raft_consensus.cc:482] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:52:11.854449 19722 raft_consensus.cc:3037] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:11.855566 19722 raft_consensus.cc:504] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } }
I20260621 07:52:11.856024 19722 leader_election.cc:288] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [CANDIDATE]: Term 1 election: Requested vote from peers 5906911d01134202bed9be4d170b2875 (127.12.12.3:42983), 1de5e565c961429ab12b8c5e25f72c9f (127.12.12.2:42563)
I20260621 07:52:11.856351 19636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3c092b9249484eaac62362586e6f3e" candidate_uuid: "85929ce3cf294504a55fd8e9ad3515ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5906911d01134202bed9be4d170b2875"
I20260621 07:52:11.856479 19636 raft_consensus.cc:3037] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:11.856482 19500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3c092b9249484eaac62362586e6f3e" candidate_uuid: "85929ce3cf294504a55fd8e9ad3515ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1de5e565c961429ab12b8c5e25f72c9f"
I20260621 07:52:11.856637 19500 raft_consensus.cc:3037] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:11.857781 19500 raft_consensus.cc:2454] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85929ce3cf294504a55fd8e9ad3515ba in term 1.
I20260621 07:52:11.857811 19636 raft_consensus.cc:2454] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85929ce3cf294504a55fd8e9ad3515ba in term 1.
I20260621 07:52:11.858093 19296 leader_election.cc:302] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1de5e565c961429ab12b8c5e25f72c9f, 85929ce3cf294504a55fd8e9ad3515ba; no voters:
I20260621 07:52:11.858291 19722 raft_consensus.cc:2781] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:11.858584 19722 raft_consensus.cc:686] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [term 1 LEADER]: Becoming Leader. State: Replica: 85929ce3cf294504a55fd8e9ad3515ba, State: Running, Role: LEADER
I20260621 07:52:11.858976 19722 consensus_queue.cc:237] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } }
I20260621 07:52:11.862941 18444 catalog_manager.cc:5495] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 85929ce3cf294504a55fd8e9ad3515ba (127.12.12.1). New cstate: current_term: 1 leader_uuid: "85929ce3cf294504a55fd8e9ad3515ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } health_report { overall_health: UNKNOWN } } }
I20260621 07:52:11.895256 19712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.891558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.12.12.1:44609)
0621 07:52:11.891708 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.891713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.891727 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:52:11.891890 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.891893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.891949 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.892064 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:52:11.892400 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.892922 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.893773 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.893997 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.894110 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.894453 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:52:11.894878 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:52:11.895013 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.895053 (+ 40us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.895108 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 07:52:11.901266 19500 raft_consensus.cc:1261] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f [term 1 FOLLOWER]: Refusing update from remote peer 85929ce3cf294504a55fd8e9ad3515ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:11.901266 19636 raft_consensus.cc:1261] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875 [term 1 FOLLOWER]: Refusing update from remote peer 85929ce3cf294504a55fd8e9ad3515ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:11.902199 19722 consensus_queue.cc:1035] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:11.902556 19727 consensus_queue.cc:1035] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:11.915827 19731 mvcc.cc:205] Tried to move back new op lower bound from 7299188047460655104 to 7299188047297777664. Current Snapshot: MvccSnapshot[applied={T|T < 7299188047460655104}]
I20260621 07:52:11.921463 18444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:11.921643 18444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:11.925622 18444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bae2da0cc2794cc78af1b0c07edafeaf: Sending DeleteTablet for 3 replicas of tablet 8c3c092b9249484eaac62362586e6f3e
I20260621 07:52:11.926405 19480 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3c092b9249484eaac62362586e6f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:11 UTC) from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:41140
I20260621 07:52:11.926599 19616 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3c092b9249484eaac62362586e6f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:11 UTC) from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:33902
I20260621 07:52:11.926748 19343 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3c092b9249484eaac62362586e6f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:11 UTC) from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:37102
I20260621 07:52:11.927178 19742 tablet_replica.cc:330] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba: stopping tablet replica
I20260621 07:52:11.927520 19742 raft_consensus.cc:2229] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:52:11.927592 19741 tablet_replica.cc:330] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875: stopping tablet replica
I20260621 07:52:11.927852 19741 raft_consensus.cc:2229] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:11.927933 19742 raft_consensus.cc:2258] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:11.928146 19741 raft_consensus.cc:2258] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:11.929108 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 18413
I20260621 07:52:11.930651 19741 ts_tablet_manager.cc:1905] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:11.931209 19742 ts_tablet_manager.cc:1905] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:11.934264 19742 ts_tablet_manager.cc:1918] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:11.934345 19741 ts_tablet_manager.cc:1918] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:11.934475 19741 log.cc:1196] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/8c3c092b9249484eaac62362586e6f3e
I20260621 07:52:11.934512 19742 log.cc:1196] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-0/wal/wals/8c3c092b9249484eaac62362586e6f3e
I20260621 07:52:11.934875 19741 ts_tablet_manager.cc:1939] T 8c3c092b9249484eaac62362586e6f3e P 5906911d01134202bed9be4d170b2875: Deleting consensus metadata
I20260621 07:52:11.935094 19742 ts_tablet_manager.cc:1939] T 8c3c092b9249484eaac62362586e6f3e P 85929ce3cf294504a55fd8e9ad3515ba: Deleting consensus metadata
W20260621 07:52:11.936318 19570 connection.cc:535] server connection from 127.0.0.1:33902 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260621 07:52:11.936350 19299 connection.cc:535] server connection from 127.0.0.1:37102 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 07:52:11.936712 19740 tablet_replica.cc:330] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f: stopping tablet replica
I20260621 07:52:11.937089 19740 raft_consensus.cc:2229] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:11.937477 19740 raft_consensus.cc:2258] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:11.942401 19740 ts_tablet_manager.cc:1905] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:11.945328 19740 ts_tablet_manager.cc:1918] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:11.945451 19740 log.cc:1196] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/8c3c092b9249484eaac62362586e6f3e
I20260621 07:52:11.945848 19740 ts_tablet_manager.cc:1939] T 8c3c092b9249484eaac62362586e6f3e P 1de5e565c961429ab12b8c5e25f72c9f: Deleting consensus metadata
W20260621 07:52:11.947213 19434 connection.cc:441] server connection from 127.0.0.1:41140 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41140 (request call id 1) could send its response
I20260621 07:52:11.947600 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.928484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.12.12.61:35893)
0621 07:52:11.928594 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.928599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.936652 (+ 8053us) server_negotiation.cc:407] Connection header received
0621 07:52:11.936962 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.936965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.937024 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.937077 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.938251 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.939112 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.940328 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.940712 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.940838 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.943364 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.943368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.943371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.943398 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.945431 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.945952 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.945956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.945958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.946041 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.946311 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.946314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.946316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.946569 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.946688 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.947279 (+ 591us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.947435 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 07:52:11.950268 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.940289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55910 (local address 127.12.12.60:44397)
0621 07:52:11.940396 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.940401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.940722 (+ 321us) server_negotiation.cc:407] Connection header received
0621 07:52:11.940891 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.940896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.940962 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.941021 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.942238 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.942949 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.944064 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.944386 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.944465 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.946181 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.946184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.946187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.946213 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.948010 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.948597 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.948601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.948603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.948694 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.949290 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.949297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.949302 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.949529 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.949637 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.949974 (+ 337us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.950083 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 07:52:11.956866 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.929103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55896 (local address 127.12.12.60:44397)
0621 07:52:11.929209 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.929215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.948694 (+ 19479us) server_negotiation.cc:407] Connection header received
0621 07:52:11.948796 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.948799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.948858 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.948949 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.949638 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.950492 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.951636 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.951994 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.952065 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.953600 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.953605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.953612 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.953640 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.955229 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.955712 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.955716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.955718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.955799 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.956063 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.956067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.956069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.956231 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.956362 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.956624 (+ 262us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.956735 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260621 07:52:11.956866 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:11.941554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.12.12.61:35893)
0621 07:52:11.941677 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:11.941684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:11.947789 (+ 6105us) server_negotiation.cc:407] Connection header received
0621 07:52:11.947995 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:11.947998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:11.948051 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:11.948108 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:11.949373 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.950108 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.951312 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:11.951611 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:11.951686 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:11.953242 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:11.953245 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:11.953248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:11.953273 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:11.954929 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.955426 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.955430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.955432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.955517 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:11.955818 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:11.955821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:11.955823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:11.956033 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:11.956157 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:11.956576 (+ 419us) server_negotiation.cc:299] Negotiation successful
0621 07:52:11.956694 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:52:12.916496 19750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:12.915740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.1:47429)
0621 07:52:12.916248 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:12.916375 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":75,"threads_started":1}
W20260621 07:52:12.916893 19410 heartbeater.cc:644] Failed to heartbeat to 127.12.12.62:39045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
I20260621 07:52:12.918218 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:12.917554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.2:49251)
0621 07:52:12.918004 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:12.918118 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":66,"threads_started":1}
I20260621 07:52:12.918449 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:12.918053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.3:32999)
0621 07:52:12.918243 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:12.918332 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
W20260621 07:52:12.918606 19547 heartbeater.cc:644] Failed to heartbeat to 127.12.12.62:39045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
W20260621 07:52:12.918931 19683 heartbeater.cc:644] Failed to heartbeat to 127.12.12.62:39045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
I20260621 07:52:13.426901 19753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bae2da0cc2794cc78af1b0c07edafeaf)
I20260621 07:52:13.427132 19753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } }
I20260621 07:52:13.427568 19754 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bae2da0cc2794cc78af1b0c07edafeaf)
I20260621 07:52:13.427755 19754 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } }
I20260621 07:52:13.428301 19753 leader_election.cc:288] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bae2da0cc2794cc78af1b0c07edafeaf (127.12.12.62:39045), d8699395d82d49d3a6312d71a6287a1c (127.12.12.60:44397)
I20260621 07:52:13.428678 19754 leader_election.cc:288] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bae2da0cc2794cc78af1b0c07edafeaf (127.12.12.62:39045), 2aeaf63573984aafa402286856776296 (127.12.12.61:35893)
I20260621 07:52:13.429011 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:13.428469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:36590)
0621 07:52:13.428823 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:13.428906 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":60,"threads_started":1}
W20260621 07:52:13.429308 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
I20260621 07:52:13.429503 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:13.429006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:36600)
0621 07:52:13.429280 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:13.429356 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":49,"threads_started":1}
W20260621 07:52:13.429775 19216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
I20260621 07:52:13.430207 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:13.429998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:36614)
0621 07:52:13.430080 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:13.430137 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260621 07:52:13.430370 19031 leader_election.cc:334] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bae2da0cc2794cc78af1b0c07edafeaf (127.12.12.62:39045): Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
I20260621 07:52:13.430804 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:13.430571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:36616)
0621 07:52:13.430679 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:13.430731 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 07:52:13.430965 19216 leader_election.cc:334] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bae2da0cc2794cc78af1b0c07edafeaf (127.12.12.62:39045): Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
I20260621 07:52:13.433406 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:13.428510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:44397 (local address 127.0.0.1:43452)
0621 07:52:13.429149 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:13.429162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:13.429309 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:13.429737 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:13.429742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:13.429765 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:13.429883 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:13.429892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:13.430964 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:13.430967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:13.432205 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:13.432217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:13.433019 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:13.433021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:13.433089 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:13.433109 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 07:52:13.433169 (+ 60us) client_negotiation.cc:240] Negotiation successful
0621 07:52:13.433222 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":91,"threads_started":1}
I20260621 07:52:13.433404 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:13.428715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.12.12.60:44397)
0621 07:52:13.428996 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:13.429001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:13.429231 (+ 230us) server_negotiation.cc:407] Connection header received
0621 07:52:13.429470 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:13.429473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:13.429523 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:13.429588 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:13.430009 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:13.430855 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:13.432330 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:13.432905 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:13.433007 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:13.433064 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:13.433194 (+ 130us) server_negotiation.cc:299] Negotiation successful
0621 07:52:13.433260 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":54,"threads_started":1}
I20260621 07:52:13.433790 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:13.428844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:35893 (local address 127.0.0.1:49958)
0621 07:52:13.429225 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:13.429239 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:13.429345 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:13.430227 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:13.430231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:13.430249 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:13.430351 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:13.430356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:13.431375 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:13.431379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:13.432537 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:13.432545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:13.433400 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:13.433404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:13.433509 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:13.433543 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 07:52:13.433595 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 07:52:13.433669 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":65,"threads_started":1}
I20260621 07:52:13.434111 19240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2aeaf63573984aafa402286856776296" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d8699395d82d49d3a6312d71a6287a1c" is_pre_election: true
I20260621 07:52:13.434248 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:13.429100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.12.12.61:35893)
0621 07:52:13.429555 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:13.429562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:13.429577 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:52:13.429618 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:13.429621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:13.429664 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:13.430472 (+ 808us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:13.430558 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:13.431269 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:13.432645 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:13.433273 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:13.433955 (+ 682us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:13.434032 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:13.434065 (+ 33us) server_negotiation.cc:299] Negotiation successful
0621 07:52:13.434133 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":60,"threads_started":1}
I20260621 07:52:13.434315 19240 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aeaf63573984aafa402286856776296 in term 1.
I20260621 07:52:13.434664 19030 leader_election.cc:302] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aeaf63573984aafa402286856776296, d8699395d82d49d3a6312d71a6287a1c; no voters: bae2da0cc2794cc78af1b0c07edafeaf
I20260621 07:52:13.434628 19055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8699395d82d49d3a6312d71a6287a1c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2aeaf63573984aafa402286856776296" is_pre_election: true
I20260621 07:52:13.434880 19055 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8699395d82d49d3a6312d71a6287a1c in term 1.
I20260621 07:52:13.434989 19753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260621 07:52:13.435045 19753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bae2da0cc2794cc78af1b0c07edafeaf)
I20260621 07:52:13.435089 19753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:52:13.435202 19217 leader_election.cc:302] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aeaf63573984aafa402286856776296, d8699395d82d49d3a6312d71a6287a1c; no voters: bae2da0cc2794cc78af1b0c07edafeaf
I20260621 07:52:13.435382 19754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260621 07:52:13.435428 19754 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 1 FOLLOWER]: Starting leader election (detected failure of leader bae2da0cc2794cc78af1b0c07edafeaf)
I20260621 07:52:13.435467 19754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:52:13.436200 19753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } }
I20260621 07:52:13.436388 19754 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } }
I20260621 07:52:13.436537 19753 leader_election.cc:288] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [CANDIDATE]: Term 2 election: Requested vote from peers bae2da0cc2794cc78af1b0c07edafeaf (127.12.12.62:39045), d8699395d82d49d3a6312d71a6287a1c (127.12.12.60:44397)
I20260621 07:52:13.436733 19754 leader_election.cc:288] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [CANDIDATE]: Term 2 election: Requested vote from peers bae2da0cc2794cc78af1b0c07edafeaf (127.12.12.62:39045), 2aeaf63573984aafa402286856776296 (127.12.12.61:35893)
I20260621 07:52:13.436884 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:13.436621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:36618)
0621 07:52:13.436766 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:13.436812 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260621 07:52:13.436896 19240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2aeaf63573984aafa402286856776296" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d8699395d82d49d3a6312d71a6287a1c"
I20260621 07:52:13.437045 19240 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2aeaf63573984aafa402286856776296 in current term 2: Already voted for candidate d8699395d82d49d3a6312d71a6287a1c in this term.
I20260621 07:52:13.437125 19055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8699395d82d49d3a6312d71a6287a1c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2aeaf63573984aafa402286856776296"
I20260621 07:52:13.437275 19055 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d8699395d82d49d3a6312d71a6287a1c in current term 2: Already voted for candidate 2aeaf63573984aafa402286856776296 in this term.
I20260621 07:52:13.437556 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:13.437142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:36624)
0621 07:52:13.437431 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:13.437478 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220}
I20260621 07:52:13.437721 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:13.437509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:36632)
0621 07:52:13.437621 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:13.437669 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 07:52:13.437824 19031 leader_election.cc:334] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bae2da0cc2794cc78af1b0c07edafeaf (127.12.12.62:39045): Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
I20260621 07:52:13.437919 19031 leader_election.cc:302] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aeaf63573984aafa402286856776296; no voters: bae2da0cc2794cc78af1b0c07edafeaf, d8699395d82d49d3a6312d71a6287a1c
I20260621 07:52:13.438063 19753 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260621 07:52:13.438630 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:13.438461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:36642)
0621 07:52:13.438546 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:13.438587 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260621 07:52:13.438733 19216 leader_election.cc:334] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bae2da0cc2794cc78af1b0c07edafeaf (127.12.12.62:39045): Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
I20260621 07:52:13.438808 19216 leader_election.cc:302] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8699395d82d49d3a6312d71a6287a1c; no voters: 2aeaf63573984aafa402286856776296, bae2da0cc2794cc78af1b0c07edafeaf
I20260621 07:52:13.438930 19754 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260621 07:52:13.918102 19763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:13.917562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.1:58343)
0621 07:52:13.917904 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:13.917978 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":118,"threads_started":1}
I20260621 07:52:13.919600 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:13.919104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.2:52969)
0621 07:52:13.919425 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:13.919499 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
I20260621 07:52:13.919965 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:13.919415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.3:43495)
0621 07:52:13.919829 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:13.919892 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
I20260621 07:52:14.919440 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:14.918926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.1:39843)
0621 07:52:14.919230 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:14.919314 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":85,"threads_started":1}
I20260621 07:52:14.920641 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:14.920236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.2:57899)
0621 07:52:14.920452 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:14.920542 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
I20260621 07:52:14.921031 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:14.920632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.3:34653)
0621 07:52:14.920899 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:14.920958 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":60,"threads_started":1}
I20260621 07:52:15.061228 19769 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:52:15.061434 19769 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } }
I20260621 07:52:15.061935 19769 leader_election.cc:288] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bae2da0cc2794cc78af1b0c07edafeaf (127.12.12.62:39045), 2aeaf63573984aafa402286856776296 (127.12.12.61:35893)
I20260621 07:52:15.062495 19055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8699395d82d49d3a6312d71a6287a1c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2aeaf63573984aafa402286856776296" is_pre_election: true
I20260621 07:52:15.062657 19055 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8699395d82d49d3a6312d71a6287a1c in term 2.
I20260621 07:52:15.062707 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.062171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:36916)
0621 07:52:15.062512 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.062572 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":70,"threads_started":1}
I20260621 07:52:15.063047 19217 leader_election.cc:302] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aeaf63573984aafa402286856776296, d8699395d82d49d3a6312d71a6287a1c; no voters:
I20260621 07:52:15.063513 19769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260621 07:52:15.063580 19769 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:52:15.063613 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.063376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:36930)
0621 07:52:15.063487 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.063546 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260621 07:52:15.063701 19216 leader_election.cc:334] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bae2da0cc2794cc78af1b0c07edafeaf (127.12.12.62:39045): Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
I20260621 07:52:15.063621 19769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 2 FOLLOWER]: Advancing to term 3
I20260621 07:52:15.064934 19769 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } }
I20260621 07:52:15.065258 19769 leader_election.cc:288] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [CANDIDATE]: Term 3 election: Requested vote from peers bae2da0cc2794cc78af1b0c07edafeaf (127.12.12.62:39045), 2aeaf63573984aafa402286856776296 (127.12.12.61:35893)
I20260621 07:52:15.065649 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.065411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:36942)
0621 07:52:15.065514 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.065584 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 07:52:15.065685 19055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8699395d82d49d3a6312d71a6287a1c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2aeaf63573984aafa402286856776296"
I20260621 07:52:15.065831 19055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 2 FOLLOWER]: Advancing to term 3
I20260621 07:52:15.066321 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.066107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:36952)
0621 07:52:15.066216 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.066266 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 07:52:15.066462 19216 leader_election.cc:334] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bae2da0cc2794cc78af1b0c07edafeaf (127.12.12.62:39045): Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
I20260621 07:52:15.067101 19055 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8699395d82d49d3a6312d71a6287a1c in term 3.
I20260621 07:52:15.067435 19217 leader_election.cc:302] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aeaf63573984aafa402286856776296, d8699395d82d49d3a6312d71a6287a1c; no voters: bae2da0cc2794cc78af1b0c07edafeaf
I20260621 07:52:15.067639 19769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 3 FOLLOWER]: Leader election won for term 3
I20260621 07:52:15.067876 19769 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [term 3 LEADER]: Becoming Leader. State: Replica: d8699395d82d49d3a6312d71a6287a1c, State: Running, Role: LEADER
I20260621 07:52:15.068219 19769 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } }
I20260621 07:52:15.069928 19772 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8699395d82d49d3a6312d71a6287a1c. Latest consensus state: current_term: 3 leader_uuid: "d8699395d82d49d3a6312d71a6287a1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } } }
I20260621 07:52:15.070080 19772 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:15.070483 19774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:52:15.072669 19774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:52:15.073030 19774 catalog_manager.cc:1259] Loaded cluster ID: d3f80f6fa24041078a598910c09aaaf3
I20260621 07:52:15.073076 19774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:52:15.073571 19774 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:52:15.073916 19774 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: Loaded TSK: 0
I20260621 07:52:15.074661 19774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:52:15.095629 19230 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:52:15.098910 19230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:52:15.101939 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.101658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:36974)
0621 07:52:15.101784 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.101847 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 07:52:15.102025 19055 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [term 3 FOLLOWER]: Refusing update from remote peer d8699395d82d49d3a6312d71a6287a1c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260621 07:52:15.102619 19772 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260621 07:52:15.102787 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.102551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.0.0.1:36978)
0621 07:52:15.102669 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.102722 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260621 07:52:15.102887 19216 consensus_peers.cc:487] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c -> Peer bae2da0cc2794cc78af1b0c07edafeaf (127.12.12.62:39045): Couldn't send request to peer bae2da0cc2794cc78af1b0c07edafeaf. Status: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260621 07:52:15.104245 19776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8699395d82d49d3a6312d71a6287a1c. Latest consensus state: current_term: 3 leader_uuid: "d8699395d82d49d3a6312d71a6287a1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } } }
I20260621 07:52:15.104414 19776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:52:15.105652 19772 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d8699395d82d49d3a6312d71a6287a1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } } }
I20260621 07:52:15.105794 19772 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:15.106278 19776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d8699395d82d49d3a6312d71a6287a1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bae2da0cc2794cc78af1b0c07edafeaf" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39045 } } peers { permanent_uuid: "2aeaf63573984aafa402286856776296" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "d8699395d82d49d3a6312d71a6287a1c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 44397 } attrs { promote: false } } }
I20260621 07:52:15.106395 19776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aeaf63573984aafa402286856776296 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:52:15.116582 19784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.112464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.12.12.2:42563)
0621 07:52:15.112963 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:15.112967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:15.112979 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:52:15.113043 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:15.113045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:15.113094 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:15.113184 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:15.113488 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:15.114033 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:15.115372 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:15.116031 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:15.116154 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:15.116223 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:15.116380 (+ 157us) server_negotiation.cc:299] Negotiation successful
0621 07:52:15.116459 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":76,"threads_started":1}
I20260621 07:52:15.116582 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.111996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:42563 (local address 127.0.0.1:52954)
0621 07:52:15.112400 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.112413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:15.112495 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:15.113199 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:15.113202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:15.113219 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:15.113341 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:15.113351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:15.114149 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:15.114155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:15.115226 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:15.115236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:15.116150 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:15.116153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:15.116234 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:15.116261 (+ 27us) client_negotiation.cc:769] Sending connection context
0621 07:52:15.116363 (+ 102us) client_negotiation.cc:240] Negotiation successful
0621 07:52:15.116434 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":53,"thread_start_us":117,"threads_started":1}
I20260621 07:52:15.117024 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.112201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:44609 (local address 127.0.0.1:45138)
0621 07:52:15.112571 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.112584 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:15.112674 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:15.113124 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:15.113130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:15.113151 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:15.113298 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:15.113305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:15.114343 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:15.114346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:15.115408 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:15.115417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:15.116376 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:15.116379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:15.116463 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:15.116483 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 07:52:15.116548 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 07:52:15.116613 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":26,"thread_start_us":88,"threads_started":1}
I20260621 07:52:15.117265 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.111759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:42983 (local address 127.0.0.1:41522)
0621 07:52:15.112035 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.112052 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:15.112147 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:15.113653 (+ 1506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:15.113656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:15.113672 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:15.113744 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:15.113749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:15.114824 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:15.114827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:15.115908 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:15.115918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:15.116876 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:15.116883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:15.116967 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:15.116991 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 07:52:15.117074 (+ 83us) client_negotiation.cc:240] Negotiation successful
0621 07:52:15.117154 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204}
I20260621 07:52:15.117448 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.111829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41522 (local address 127.12.12.3:42983)
0621 07:52:15.113413 (+ 1584us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:15.113427 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:15.113440 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:52:15.113489 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:15.113492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:15.113535 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:15.113843 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:15.113944 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:15.114710 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:15.116034 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:15.116657 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:15.116860 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:15.116922 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:15.117105 (+ 183us) server_negotiation.cc:299] Negotiation successful
0621 07:52:15.117205 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1492,"thread_start_us":75,"threads_started":1}
I20260621 07:52:15.117498 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.112558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.12.12.1:44609)
0621 07:52:15.112828 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:15.112833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:15.112850 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:52:15.112903 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:15.112908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:15.112959 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:15.113045 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:15.113433 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:15.114221 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:15.115540 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:15.116167 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:15.116264 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:15.116322 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:15.117267 (+ 945us) server_negotiation.cc:299] Negotiation successful
0621 07:52:15.117347 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260621 07:52:15.117667 19480 tablet_service.cc:1467] Processing CreateTablet for tablet dc9de8920ce4483b848857a415ca0e9f (DEFAULT_TABLE table=test-table [id=9cca241a2e444815828378e020e6cde3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:15.117969 19480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9de8920ce4483b848857a415ca0e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:15.117913 19616 tablet_service.cc:1467] Processing CreateTablet for tablet dc9de8920ce4483b848857a415ca0e9f (DEFAULT_TABLE table=test-table [id=9cca241a2e444815828378e020e6cde3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:15.118083 19343 tablet_service.cc:1467] Processing CreateTablet for tablet dc9de8920ce4483b848857a415ca0e9f (DEFAULT_TABLE table=test-table [id=9cca241a2e444815828378e020e6cde3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:15.118145 19616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9de8920ce4483b848857a415ca0e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:15.118330 19343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9de8920ce4483b848857a415ca0e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:15.120507 19786 tablet_bootstrap.cc:492] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875: Bootstrap starting.
I20260621 07:52:15.120925 19787 tablet_bootstrap.cc:492] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba: Bootstrap starting.
I20260621 07:52:15.121148 19788 tablet_bootstrap.cc:492] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f: Bootstrap starting.
I20260621 07:52:15.121613 19786 tablet_bootstrap.cc:654] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:15.122004 19788 tablet_bootstrap.cc:654] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:15.122890 19786 tablet_bootstrap.cc:492] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875: No bootstrap required, opened a new log
I20260621 07:52:15.122998 19786 ts_tablet_manager.cc:1397] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 07:52:15.123248 19788 tablet_bootstrap.cc:492] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f: No bootstrap required, opened a new log
I20260621 07:52:15.123322 19787 tablet_bootstrap.cc:654] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:15.123348 19788 ts_tablet_manager.cc:1397] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:52:15.123389 19786 raft_consensus.cc:348] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } }
I20260621 07:52:15.123512 19786 raft_consensus.cc:374] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:15.123539 19786 raft_consensus.cc:729] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5906911d01134202bed9be4d170b2875, State: Initialized, Role: FOLLOWER
I20260621 07:52:15.123678 19786 consensus_queue.cc:260] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } }
I20260621 07:52:15.123757 19788 raft_consensus.cc:348] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } }
I20260621 07:52:15.123888 19788 raft_consensus.cc:374] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:15.123924 19788 raft_consensus.cc:729] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1de5e565c961429ab12b8c5e25f72c9f, State: Initialized, Role: FOLLOWER
I20260621 07:52:15.124058 19786 ts_tablet_manager.cc:1428] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:52:15.124056 19788 consensus_queue.cc:260] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } }
I20260621 07:52:15.124404 19788 ts_tablet_manager.cc:1428] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:52:15.125084 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.124804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.2:48979)
0621 07:52:15.124951 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.125010 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260621 07:52:15.125496 19787 tablet_bootstrap.cc:492] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba: No bootstrap required, opened a new log
I20260621 07:52:15.125605 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.125245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.3:41391)
0621 07:52:15.125492 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.125544 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260621 07:52:15.125628 19787 ts_tablet_manager.cc:1397] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260621 07:52:15.126056 19787 raft_consensus.cc:348] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } }
I20260621 07:52:15.126209 19787 raft_consensus.cc:374] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:15.126264 19787 raft_consensus.cc:729] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85929ce3cf294504a55fd8e9ad3515ba, State: Initialized, Role: FOLLOWER
I20260621 07:52:15.126461 19787 consensus_queue.cc:260] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } }
I20260621 07:52:15.126875 19787 ts_tablet_manager.cc:1428] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:52:15.127672 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.127432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.1:53169)
0621 07:52:15.127533 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.127606 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260621 07:52:15.127954 19546 heartbeater.cc:503] Master 127.12.12.60:44397 was elected leader, sending a full tablet report...
I20260621 07:52:15.128319 19682 heartbeater.cc:503] Master 127.12.12.60:44397 was elected leader, sending a full tablet report...
I20260621 07:52:15.128800 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.128119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.3:53867)
0621 07:52:15.128676 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.128729 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494}
I20260621 07:52:15.129210 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.128955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.2:33285)
0621 07:52:15.129084 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.129141 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260621 07:52:15.129912 19409 heartbeater.cc:503] Master 127.12.12.60:44397 was elected leader, sending a full tablet report...
I20260621 07:52:15.131441 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.131207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.1:34249)
0621 07:52:15.131331 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.131388 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260621 07:52:15.146153 19794 raft_consensus.cc:482] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:52:15.146297 19794 raft_consensus.cc:504] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } }
I20260621 07:52:15.146715 19794 leader_election.cc:288] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5906911d01134202bed9be4d170b2875 (127.12.12.3:42983), 1de5e565c961429ab12b8c5e25f72c9f (127.12.12.2:42563)
I20260621 07:52:15.147284 19500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9de8920ce4483b848857a415ca0e9f" candidate_uuid: "85929ce3cf294504a55fd8e9ad3515ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1de5e565c961429ab12b8c5e25f72c9f" is_pre_election: true
I20260621 07:52:15.147303 19636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9de8920ce4483b848857a415ca0e9f" candidate_uuid: "85929ce3cf294504a55fd8e9ad3515ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5906911d01134202bed9be4d170b2875" is_pre_election: true
I20260621 07:52:15.147461 19500 raft_consensus.cc:2454] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85929ce3cf294504a55fd8e9ad3515ba in term 0.
I20260621 07:52:15.147485 19636 raft_consensus.cc:2454] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85929ce3cf294504a55fd8e9ad3515ba in term 0.
I20260621 07:52:15.147851 19296 leader_election.cc:302] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1de5e565c961429ab12b8c5e25f72c9f, 85929ce3cf294504a55fd8e9ad3515ba; no voters:
I20260621 07:52:15.148042 19794 raft_consensus.cc:2781] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:52:15.148104 19794 raft_consensus.cc:482] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:52:15.148144 19794 raft_consensus.cc:3037] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:15.149276 19794 raft_consensus.cc:504] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } }
I20260621 07:52:15.149659 19794 leader_election.cc:288] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [CANDIDATE]: Term 1 election: Requested vote from peers 5906911d01134202bed9be4d170b2875 (127.12.12.3:42983), 1de5e565c961429ab12b8c5e25f72c9f (127.12.12.2:42563)
I20260621 07:52:15.150064 19500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9de8920ce4483b848857a415ca0e9f" candidate_uuid: "85929ce3cf294504a55fd8e9ad3515ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1de5e565c961429ab12b8c5e25f72c9f"
I20260621 07:52:15.150065 19636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9de8920ce4483b848857a415ca0e9f" candidate_uuid: "85929ce3cf294504a55fd8e9ad3515ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5906911d01134202bed9be4d170b2875"
I20260621 07:52:15.150187 19500 raft_consensus.cc:3037] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:15.150209 19636 raft_consensus.cc:3037] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:15.151218 19500 raft_consensus.cc:2454] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85929ce3cf294504a55fd8e9ad3515ba in term 1.
I20260621 07:52:15.151391 19636 raft_consensus.cc:2454] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85929ce3cf294504a55fd8e9ad3515ba in term 1.
I20260621 07:52:15.151554 19296 leader_election.cc:302] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1de5e565c961429ab12b8c5e25f72c9f, 85929ce3cf294504a55fd8e9ad3515ba; no voters:
I20260621 07:52:15.151726 19794 raft_consensus.cc:2781] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:15.151950 19794 raft_consensus.cc:686] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [term 1 LEADER]: Becoming Leader. State: Replica: 85929ce3cf294504a55fd8e9ad3515ba, State: Running, Role: LEADER
I20260621 07:52:15.152678 19794 consensus_queue.cc:237] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } }
I20260621 07:52:15.152915 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.152575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.1:46355)
0621 07:52:15.152734 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.152806 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 07:52:15.155122 19230 catalog_manager.cc:5495] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 85929ce3cf294504a55fd8e9ad3515ba (127.12.12.1). New cstate: current_term: 1 leader_uuid: "85929ce3cf294504a55fd8e9ad3515ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85929ce3cf294504a55fd8e9ad3515ba" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 44609 } health_report { overall_health: HEALTHY } } }
I20260621 07:52:15.170398 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.167204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.12.12.1:44609)
0621 07:52:15.167380 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:15.167384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:15.167397 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:52:15.167438 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:15.167440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:15.167486 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:15.167551 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:52:15.167774 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:15.168461 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:15.169281 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:15.169547 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:15.169648 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:15.169923 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:52:15.170005 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:52:15.170085 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:15.170198 (+ 113us) server_negotiation.cc:299] Negotiation successful
0621 07:52:15.170250 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 07:52:15.172490 19500 raft_consensus.cc:1261] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f [term 1 FOLLOWER]: Refusing update from remote peer 85929ce3cf294504a55fd8e9ad3515ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:15.172498 19636 raft_consensus.cc:1261] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875 [term 1 FOLLOWER]: Refusing update from remote peer 85929ce3cf294504a55fd8e9ad3515ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:15.172920 19795 consensus_queue.cc:1035] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5906911d01134202bed9be4d170b2875" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:15.173043 19794 consensus_queue.cc:1035] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "1de5e565c961429ab12b8c5e25f72c9f" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 42563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:15.174850 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.174551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.2:36963)
0621 07:52:15.174716 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.174777 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260621 07:52:15.175112 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.174841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.3:49457)
0621 07:52:15.174989 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.175042 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260621 07:52:15.178539 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.178224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.1:54673)
0621 07:52:15.178380 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.178467 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260621 07:52:15.179955 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.179626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.3:48579)
0621 07:52:15.179830 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.179887 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260621 07:52:15.180236 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.179897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.2:41829)
0621 07:52:15.180111 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.180154 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260621 07:52:15.180567 19798 mvcc.cc:205] Tried to move back new op lower bound from 7299188060862296064 to 7299188060787273728. Current Snapshot: MvccSnapshot[applied={T|T < 7299188060862296064}]
I20260621 07:52:15.181527 19800 mvcc.cc:205] Tried to move back new op lower bound from 7299188060862296064 to 7299188060787273728. Current Snapshot: MvccSnapshot[applied={T|T < 7299188060862296064 or (T in {7299188060862296064})}]
I20260621 07:52:15.182081 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.179964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39045 (local address 127.12.12.1:42271)
0621 07:52:15.180507 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.180558 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479}
I20260621 07:52:15.187201 19228 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:15.187386 19228 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:15.191398 19228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8699395d82d49d3a6312d71a6287a1c: Sending DeleteTablet for 3 replicas of tablet dc9de8920ce4483b848857a415ca0e9f
I20260621 07:52:15.192638 19480 tablet_service.cc:1514] Processing DeleteTablet for tablet dc9de8920ce4483b848857a415ca0e9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:15 UTC) from {username='oryx', principal='oryx/127.12.12.60@KRBTEST.COM'} at 127.0.0.1:52954
I20260621 07:52:15.192834 19616 tablet_service.cc:1514] Processing DeleteTablet for tablet dc9de8920ce4483b848857a415ca0e9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:15 UTC) from {username='oryx', principal='oryx/127.12.12.60@KRBTEST.COM'} at 127.0.0.1:41522
I20260621 07:52:15.193123 19806 tablet_replica.cc:330] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f: stopping tablet replica
I20260621 07:52:15.193177 19807 tablet_replica.cc:330] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875: stopping tablet replica
I20260621 07:52:15.193166 19343 tablet_service.cc:1514] Processing DeleteTablet for tablet dc9de8920ce4483b848857a415ca0e9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:15 UTC) from {username='oryx', principal='oryx/127.12.12.60@KRBTEST.COM'} at 127.0.0.1:45138
I20260621 07:52:15.193306 19807 raft_consensus.cc:2229] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:15.193336 19806 raft_consensus.cc:2229] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:15.193460 19807 raft_consensus.cc:2258] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:15.193595 19806 raft_consensus.cc:2258] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:15.193797 19808 tablet_replica.cc:330] T dc9de8920ce4483b848857a415ca0e9f P 85929ce3cf294504a55fd8e9ad3515ba: stopping tablet replica
I20260621 07:52:15.193953 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 19189
I20260621 07:52:15.195451 19806 ts_tablet_manager.cc:1905] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:15.198191 19806 ts_tablet_manager.cc:1918] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:15.198287 19806 log.cc:1196] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/dc9de8920ce4483b848857a415ca0e9f
I20260621 07:52:15.198603 19806 ts_tablet_manager.cc:1939] T dc9de8920ce4483b848857a415ca0e9f P 1de5e565c961429ab12b8c5e25f72c9f: Deleting consensus metadata
I20260621 07:52:15.198837 19807 ts_tablet_manager.cc:1905] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:15.199488 19214 catalog_manager.cc:4841] TS 1de5e565c961429ab12b8c5e25f72c9f (127.12.12.2:42563): tablet dc9de8920ce4483b848857a415ca0e9f (table test-table [id=9cca241a2e444815828378e020e6cde3]) successfully deleted
I20260621 07:52:15.200837 19807 ts_tablet_manager.cc:1918] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:15.200909 19807 log.cc:1196] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/dc9de8920ce4483b848857a415ca0e9f
I20260621 07:52:15.201113 19807 ts_tablet_manager.cc:1939] T dc9de8920ce4483b848857a415ca0e9f P 5906911d01134202bed9be4d170b2875: Deleting consensus metadata
I20260621 07:52:15.201979 19214 catalog_manager.cc:4841] TS 5906911d01134202bed9be4d170b2875 (127.12.12.3:42983): tablet dc9de8920ce4483b848857a415ca0e9f (table test-table [id=9cca241a2e444815828378e020e6cde3]) successfully deleted
I20260621 07:52:15.207839 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 19417
I20260621 07:52:15.208721 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:15.208061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:44609 (local address 127.0.0.1:45144)
0621 07:52:15.208597 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:15.208645 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:44609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461}
W20260621 07:52:15.208945 19216 catalog_manager.cc:4584] TS 85929ce3cf294504a55fd8e9ad3515ba (127.12.12.1:44609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc9de8920ce4483b848857a415ca0e9f: Network error: Client connection negotiation failed: client connection to 127.12.12.1:44609: connect: Connection refused (error 111)
I20260621 07:52:15.214103 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 19553
I20260621 07:52:15.220055 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 18909
I20260621 07:52:15.225956 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 19009
2026-06-21T07:52:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6924 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:52:15 dist-test-slave-lbpt krb5kdc[19811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:52:15 dist-test-slave-lbpt krb5kdc[19811](info): set up 2 sockets
Jun 21 07:52:15 dist-test-slave-lbpt krb5kdc[19811](info): commencing operation
krb5kdc: starting...
W20260621 07:52:16.275118 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:52:16 dist-test-slave-lbpt krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:52:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.62.keytab.
Entry for principal oryx/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.62.keytab.
I20260621 07:52:16.401980 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:43865
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.62.keytab
--principal=oryx/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:43951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:43865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:16.512475 19824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:16.512825 19824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:16.512908 19824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:16.516989 19824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:16.517092 19824 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 07:52:16.517122 19824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:16.517165 19824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:16.517223 19824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:16.517263 19824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:52:16.522768 19824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:43951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:43865
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.62.keytab
--principal=oryx/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:43865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:16.523940 19824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:16.525000 19824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:16.533691 19832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:16.533684 19830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:16.533684 19829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:16.534163 19824 server_base.cc:1034] running on GCE node
I20260621 07:52:16.534797 19824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:16.535909 19824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:16.537106 19824 hybrid_clock.cc:648] HybridClock initialized: now 1782028336537098 us; error 33 us; skew 500 ppm
Jun 21 07:52:16 dist-test-slave-lbpt krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028336, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:16.539952 19824 init.cc:377] Logged in from keytab as oryx/127.12.12.62@KRBTEST.COM (short username oryx)
I20260621 07:52:16.540835 19824 webserver.cc:460] Webserver started at http://127.12.12.62:36469/ using document root <none> and password file <none>
I20260621 07:52:16.541360 19824 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:16.541419 19824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:16.541584 19824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:16.543782 19824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "d942cdbfceb14a9ea6522ec4d98a191c"
format_stamp: "Formatted at 2026-06-21 07:52:16 on dist-test-slave-lbpt"
server_key: "6e162bd065cd6ef50026118c4f8e0912"
server_key_iv: "3188800cf2d0ebf43942d3bebfb9b14c"
server_key_version: "encryptionkey@0"
I20260621 07:52:16.544289 19824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "d942cdbfceb14a9ea6522ec4d98a191c"
format_stamp: "Formatted at 2026-06-21 07:52:16 on dist-test-slave-lbpt"
server_key: "6e162bd065cd6ef50026118c4f8e0912"
server_key_iv: "3188800cf2d0ebf43942d3bebfb9b14c"
server_key_version: "encryptionkey@0"
I20260621 07:52:16.547964 19824 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:52:16.550547 19839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:16.551910 19824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:16.552049 19824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "d942cdbfceb14a9ea6522ec4d98a191c"
format_stamp: "Formatted at 2026-06-21 07:52:16 on dist-test-slave-lbpt"
server_key: "6e162bd065cd6ef50026118c4f8e0912"
server_key_iv: "3188800cf2d0ebf43942d3bebfb9b14c"
server_key_version: "encryptionkey@0"
I20260621 07:52:16.552166 19824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:16.576686 19824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:16.579989 19824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:16.580199 19824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:16.588413 19824 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:43865
I20260621 07:52:16.588423 19901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:43865 every 8 connection(s)
I20260621 07:52:16.589476 19824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:52:16.592337 19902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:16.598174 19902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c: Bootstrap starting.
I20260621 07:52:16.599056 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 19824
I20260621 07:52:16.599221 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:52:16.599570 12336 external_mini_cluster.cc:1428] Setting key 443c01fa4fe744df2a0c3ba665a42338
I20260621 07:52:16.601583 19902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:16.602382 19902 log.cc:824] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:16.604341 19902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c: No bootstrap required, opened a new log
I20260621 07:52:16.608191 19902 raft_consensus.cc:348] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d942cdbfceb14a9ea6522ec4d98a191c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43865 } }
I20260621 07:52:16.608505 19902 raft_consensus.cc:374] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:16.608562 19902 raft_consensus.cc:729] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d942cdbfceb14a9ea6522ec4d98a191c, State: Initialized, Role: FOLLOWER
Jun 21 07:52:16 dist-test-slave-lbpt krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM
I20260621 07:52:16.609099 19902 consensus_queue.cc:260] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d942cdbfceb14a9ea6522ec4d98a191c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43865 } }
I20260621 07:52:16.609277 19902 raft_consensus.cc:388] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:52:16.609355 19902 raft_consensus.cc:482] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:52:16.609493 19902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:16.610769 19902 raft_consensus.cc:504] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d942cdbfceb14a9ea6522ec4d98a191c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43865 } }
I20260621 07:52:16.611150 19902 leader_election.cc:302] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d942cdbfceb14a9ea6522ec4d98a191c; no voters:
I20260621 07:52:16.611474 19902 leader_election.cc:288] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:52:16.611593 19907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:16.611821 19907 raft_consensus.cc:686] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [term 1 LEADER]: Becoming Leader. State: Replica: d942cdbfceb14a9ea6522ec4d98a191c, State: Running, Role: LEADER
I20260621 07:52:16.612126 19907 consensus_queue.cc:237] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d942cdbfceb14a9ea6522ec4d98a191c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43865 } }
I20260621 07:52:16.612876 19902 sys_catalog.cc:564] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:52:16.613758 19909 sys_catalog.cc:455] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d942cdbfceb14a9ea6522ec4d98a191c. Latest consensus state: current_term: 1 leader_uuid: "d942cdbfceb14a9ea6522ec4d98a191c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d942cdbfceb14a9ea6522ec4d98a191c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43865 } } }
I20260621 07:52:16.613832 19908 sys_catalog.cc:455] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d942cdbfceb14a9ea6522ec4d98a191c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d942cdbfceb14a9ea6522ec4d98a191c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43865 } } }
I20260621 07:52:16.613992 19908 sys_catalog.cc:458] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:16.614063 19909 sys_catalog.cc:458] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:16.614380 19916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:52:16.615665 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:16.601024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.12.12.62:43865)
0621 07:52:16.601529 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:16.601554 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:16.601591 (+ 37us) server_negotiation.cc:407] Connection header received
0621 07:52:16.602337 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:16.602358 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:16.602835 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:16.603191 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:16.604013 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:16.605058 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:16.605905 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:16.606375 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:16.606618 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:16.609518 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:16.609524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:16.609538 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:16.609587 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:16.611881 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:16.612382 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:16.612389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:16.612395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:16.612543 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:16.612878 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:16.612881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:16.612884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:16.613419 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:16.613604 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:16.614325 (+ 721us) server_negotiation.cc:299] Negotiation successful
0621 07:52:16.614653 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":169,"threads_started":1}
I20260621 07:52:16.617771 19916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:52:16.623446 19916 catalog_manager.cc:1347] Generated new cluster ID: d35de2d6b8c647f490eedbe773c17861
I20260621 07:52:16.623526 19916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:52:16.643266 19916 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:52:16.644776 19916 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:52:16.653847 19916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c: Generated new TSK 0
I20260621 07:52:16.654739 19916 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.1.keytab.
Entry for principal oryx/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.1.keytab.
I20260621 07:52:16.685242 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43865
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.1.keytab
--principal=oryx/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:43951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:16.814633 19928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:16.814925 19928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:16.814972 19928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:16.819128 19928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:16.819207 19928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:16.819249 19928 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:52:16.819284 19928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:52:16.824782 19928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:43951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.1.keytab
--principal=oryx/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.62:43865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:16.825958 19928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:16.827127 19928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:16.837095 19933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:16.837142 19934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:16.837165 19936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:16.837572 19928 server_base.cc:1034] running on GCE node
I20260621 07:52:16.838042 19928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:16.838797 19928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:16.840010 19928 hybrid_clock.cc:648] HybridClock initialized: now 1782028336839989 us; error 33 us; skew 500 ppm
Jun 21 07:52:16 dist-test-slave-lbpt krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028336, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:16.843070 19928 init.cc:377] Logged in from keytab as oryx/127.12.12.1@KRBTEST.COM (short username oryx)
I20260621 07:52:16.844159 19928 webserver.cc:460] Webserver started at http://127.12.12.1:35589/ using document root <none> and password file <none>
I20260621 07:52:16.844694 19928 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:16.844754 19928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:16.844931 19928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:16.847141 19928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "da12b3e95de94515857e94d262a8b165"
format_stamp: "Formatted at 2026-06-21 07:52:16 on dist-test-slave-lbpt"
server_key: "a1fb6481317764c2f72f6208ce50b6b1"
server_key_iv: "84ea87cde054cca7f2cd2129fdaec1ff"
server_key_version: "encryptionkey@0"
I20260621 07:52:16.847673 19928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "da12b3e95de94515857e94d262a8b165"
format_stamp: "Formatted at 2026-06-21 07:52:16 on dist-test-slave-lbpt"
server_key: "a1fb6481317764c2f72f6208ce50b6b1"
server_key_iv: "84ea87cde054cca7f2cd2129fdaec1ff"
server_key_version: "encryptionkey@0"
I20260621 07:52:16.852270 19928 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260621 07:52:16.858731 19943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:16.860144 19928 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260621 07:52:16.860271 19928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "da12b3e95de94515857e94d262a8b165"
format_stamp: "Formatted at 2026-06-21 07:52:16 on dist-test-slave-lbpt"
server_key: "a1fb6481317764c2f72f6208ce50b6b1"
server_key_iv: "84ea87cde054cca7f2cd2129fdaec1ff"
server_key_version: "encryptionkey@0"
I20260621 07:52:16.860378 19928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:16.870954 19928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:16.873984 19928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:16.874147 19928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:16.875617 19928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:16.875705 19928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:16.875766 19928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:16.875820 19928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:52:16 dist-test-slave-lbpt krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028336, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM
I20260621 07:52:16.887240 19928 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:42795
I20260621 07:52:16.887300 20058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:42795 every 8 connection(s)
I20260621 07:52:16.888520 19928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:52:16.891094 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:16.876549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:41459 (local address 127.12.12.62:43865)
0621 07:52:16.876677 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:16.876681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:16.877860 (+ 1179us) server_negotiation.cc:407] Connection header received
0621 07:52:16.879120 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:16.879124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:16.879185 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:16.879305 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:16.880867 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:16.881773 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:16.883380 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:16.883640 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:16.883777 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:16.886972 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:16.886978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:16.886982 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:16.887020 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:16.888767 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:16.889382 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:16.889387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:16.889392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:16.889506 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:16.889851 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:16.889854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:16.889855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:16.890043 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:16.890120 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:16.890852 (+ 732us) server_negotiation.cc:299] Negotiation successful
0621 07:52:16.890970 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:52:16.892063 19952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:16.877144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43865 (local address 127.12.12.1:41459)
0621 07:52:16.877662 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:16.877706 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:16.878891 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:16.879494 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:16.879505 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:16.879997 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:16.880692 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:16.880717 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:16.881906 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:16.881910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:16.883252 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:16.883262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:16.883834 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:16.883838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:16.884035 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:16.884755 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:16.884771 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:16.886724 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:16.888916 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:16.888924 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:16.888941 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:16.889269 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:16.889601 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:16.889606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:16.889608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:16.889758 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:16.890145 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:16.890150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:16.890580 (+ 430us) client_negotiation.cc:769] Sending connection context
0621 07:52:16.890790 (+ 210us) client_negotiation.cc:240] Negotiation successful
0621 07:52:16.891072 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":132,"threads_started":1}
I20260621 07:52:16.893149 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 19928
I20260621 07:52:16.893293 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:52:16.893497 20059 heartbeater.cc:347] Connected to a master server at 127.12.12.62:43865
I20260621 07:52:16.893682 12336 external_mini_cluster.cc:1428] Setting key 8bd14eab1b5d4ee8dd054822e47a9c9b
I20260621 07:52:16.893731 20059 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:16.894338 20059 heartbeater.cc:511] Master 127.12.12.62:43865 requested a full tablet report, sending...
I20260621 07:52:16.896011 19855 ts_manager.cc:194] Registered new tserver with Master: da12b3e95de94515857e94d262a8b165 (127.12.12.1:42795)
I20260621 07:52:16.897619 19855 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:41459
WARNING: no policy specified for oryx/127.12.12.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.2@KRBTEST.COM" created.
I20260621 07:52:16.910224 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:16.897628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43865 (local address 127.12.12.1:57377)
0621 07:52:16.902555 (+ 4927us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:16.902572 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:16.902727 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:16.903005 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:16.903008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:16.903225 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:16.903333 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:16.903338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:16.904206 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:16.904209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:16.904906 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:16.904915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:16.905466 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:16.905469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:16.905550 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:16.906135 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:16.906162 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:16.906621 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:16.908572 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:16.908575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:16.908580 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:16.908855 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:16.909154 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:16.909169 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:16.909173 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:16.909296 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:16.909650 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:16.909653 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:16.909754 (+ 101us) client_negotiation.cc:769] Sending connection context
0621 07:52:16.909864 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 07:52:16.910026 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":109,"threads_started":1}
I20260621 07:52:16.910220 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:16.897707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:57377 (local address 127.12.12.62:43865)
0621 07:52:16.897855 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:16.897861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:16.902665 (+ 4804us) server_negotiation.cc:407] Connection header received
0621 07:52:16.902832 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:16.902835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:16.902888 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:16.902977 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:16.903461 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:16.904074 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:16.905061 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:16.905285 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:16.905460 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:16.906766 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:16.906772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:16.906778 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:16.906808 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:16.908440 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:16.908962 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:16.908964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:16.908966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:16.909060 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:16.909377 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:16.909379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:16.909381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:16.909547 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:16.909662 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:16.909916 (+ 254us) server_negotiation.cc:299] Negotiation successful
0621 07:52:16.910049 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.2.keytab.
Entry for principal oryx/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.2.keytab.
I20260621 07:52:16.923877 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43865
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.2.keytab
--principal=oryx/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:43951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:17.039539 20068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:17.039881 20068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:17.039968 20068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:17.044183 20068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:17.044299 20068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:17.044394 20068 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:52:17.044482 20068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:52:17.050177 20068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:43951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.2.keytab
--principal=oryx/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.62:43865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:17.051371 20068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:17.052407 20068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:17.062556 20076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:17.062551 20073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:17.062551 20074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:17.063055 20068 server_base.cc:1034] running on GCE node
I20260621 07:52:17.063551 20068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:17.064361 20068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:17.065585 20068 hybrid_clock.cc:648] HybridClock initialized: now 1782028337065541 us; error 55 us; skew 500 ppm
Jun 21 07:52:17 dist-test-slave-lbpt krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028337, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:17.068692 20068 init.cc:377] Logged in from keytab as oryx/127.12.12.2@KRBTEST.COM (short username oryx)
I20260621 07:52:17.069590 20068 webserver.cc:460] Webserver started at http://127.12.12.2:38407/ using document root <none> and password file <none>
I20260621 07:52:17.070147 20068 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:17.070209 20068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:17.070537 20068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:17.072664 20068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "de4777f89fad4de0aac931ffd1538342"
format_stamp: "Formatted at 2026-06-21 07:52:17 on dist-test-slave-lbpt"
server_key: "61603dcb6ce932a30adf59b9961c8863"
server_key_iv: "fadde6508c2df780684fcd9a8b232680"
server_key_version: "encryptionkey@0"
I20260621 07:52:17.073266 20068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "de4777f89fad4de0aac931ffd1538342"
format_stamp: "Formatted at 2026-06-21 07:52:17 on dist-test-slave-lbpt"
server_key: "61603dcb6ce932a30adf59b9961c8863"
server_key_iv: "fadde6508c2df780684fcd9a8b232680"
server_key_version: "encryptionkey@0"
I20260621 07:52:17.077306 20068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:52:17.080170 20083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:17.081624 20068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:52:17.081772 20068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "de4777f89fad4de0aac931ffd1538342"
format_stamp: "Formatted at 2026-06-21 07:52:17 on dist-test-slave-lbpt"
server_key: "61603dcb6ce932a30adf59b9961c8863"
server_key_iv: "fadde6508c2df780684fcd9a8b232680"
server_key_version: "encryptionkey@0"
I20260621 07:52:17.081893 20068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:17.109535 20068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:17.113330 20068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:17.113595 20068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:17.115515 20068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:17.115569 20068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:17.115639 20068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:17.115671 20068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:52:17 dist-test-slave-lbpt krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028337, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM
I20260621 07:52:17.126693 20068 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:40171
I20260621 07:52:17.126904 20198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:40171 every 8 connection(s)
I20260621 07:52:17.127677 20068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:52:17.131407 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 20068
I20260621 07:52:17.131527 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:52:17.131598 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.116539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:36081 (local address 127.12.12.62:43865)
0621 07:52:17.116667 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:17.116670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:17.117753 (+ 1083us) server_negotiation.cc:407] Connection header received
0621 07:52:17.119038 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:17.119042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:17.119099 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:17.119216 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:17.121042 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.121953 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.123809 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.124031 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.124123 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.127459 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:17.127471 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:17.127473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:17.127500 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:17.129173 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:17.129782 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:17.129785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:17.129787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:17.129872 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:17.130206 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:17.130208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:17.130209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:17.130554 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:17.130643 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:17.131200 (+ 557us) server_negotiation.cc:299] Negotiation successful
0621 07:52:17.131407 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:52:17.131877 12336 external_mini_cluster.cc:1428] Setting key 4b4a17e146c3188920f57393bc36a249
I20260621 07:52:17.132462 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.117114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43865 (local address 127.12.12.2:36081)
0621 07:52:17.117600 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:17.117645 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:17.118715 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:17.119437 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:17.119448 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:17.119978 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:17.120786 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.120811 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.122079 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.122084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.123677 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.123690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.124203 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.124207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.124461 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.125222 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:17.125247 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:17.127210 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:17.129292 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:17.129297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:17.129353 (+ 56us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:17.129613 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:17.129969 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:17.129971 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:17.129973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:17.130109 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:17.130666 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:17.130672 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:17.130928 (+ 256us) client_negotiation.cc:769] Sending connection context
0621 07:52:17.131081 (+ 153us) client_negotiation.cc:240] Negotiation successful
0621 07:52:17.131356 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":129,"threads_started":1}
I20260621 07:52:17.133818 20199 heartbeater.cc:347] Connected to a master server at 127.12.12.62:43865
I20260621 07:52:17.133991 20199 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:17.134585 20199 heartbeater.cc:511] Master 127.12.12.62:43865 requested a full tablet report, sending...
I20260621 07:52:17.135912 19854 ts_manager.cc:194] Registered new tserver with Master: de4777f89fad4de0aac931ffd1538342 (127.12.12.2:40171)
I20260621 07:52:17.136786 19854 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:36081
WARNING: no policy specified for oryx/127.12.12.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.3@KRBTEST.COM" created.
I20260621 07:52:17.145907 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.136403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43865 (local address 127.12.12.2:41527)
0621 07:52:17.137266 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:17.137284 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:17.137426 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:17.137798 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:17.137814 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:17.138073 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:17.138234 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.138245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.139472 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.139479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.140547 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.140560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.141052 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.141056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.141161 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.141735 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:17.141752 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:17.142226 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:17.144100 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:17.144102 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:17.144104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:17.144294 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:17.144644 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:17.144649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:17.144654 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:17.144756 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:17.145238 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:17.145244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:17.145357 (+ 113us) client_negotiation.cc:769] Sending connection context
0621 07:52:17.145519 (+ 162us) client_negotiation.cc:240] Negotiation successful
0621 07:52:17.145722 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":100,"threads_started":1}
I20260621 07:52:17.145900 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.136460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:41527 (local address 127.12.12.62:43865)
0621 07:52:17.136578 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:17.136583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:17.137353 (+ 770us) server_negotiation.cc:407] Connection header received
0621 07:52:17.137530 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:17.137536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:17.137590 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:17.137645 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:17.138386 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.139284 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.140686 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.140914 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.141008 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.142378 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:17.142381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:17.142383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:17.142406 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:17.144003 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:17.144395 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:17.144397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:17.144399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:17.144520 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:17.144872 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:17.144875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:17.144876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:17.145089 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:17.145170 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:17.145590 (+ 420us) server_negotiation.cc:299] Negotiation successful
0621 07:52:17.145713 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.3.keytab.
Entry for principal oryx/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.3.keytab.
I20260621 07:52:17.162207 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43865
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.3.keytab
--principal=oryx/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:43951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:17.277961 20208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:17.278311 20208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:17.278451 20208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:17.282765 20208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:17.282876 20208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:17.282949 20208 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:52:17.283030 20208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:52:17.288841 20208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:43951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.3.keytab
--principal=oryx/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.62:43865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:17.290010 20208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:17.291194 20208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:17.302106 20216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:17.302106 20213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:17.302664 20208 server_base.cc:1034] running on GCE node
W20260621 07:52:17.304206 20214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:17.304814 20208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:17.305713 20208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:17.306908 20208 hybrid_clock.cc:648] HybridClock initialized: now 1782028337306872 us; error 55 us; skew 500 ppm
Jun 21 07:52:17 dist-test-slave-lbpt krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028337, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:17.310009 20208 init.cc:377] Logged in from keytab as oryx/127.12.12.3@KRBTEST.COM (short username oryx)
I20260621 07:52:17.311022 20208 webserver.cc:460] Webserver started at http://127.12.12.3:41897/ using document root <none> and password file <none>
I20260621 07:52:17.311604 20208 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:17.311697 20208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:17.311937 20208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:17.314222 20208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8"
format_stamp: "Formatted at 2026-06-21 07:52:17 on dist-test-slave-lbpt"
server_key: "b518ebbb4285504037dd9263f1113b44"
server_key_iv: "1517d2d40d7e1dbc00c9ffde495eb5cf"
server_key_version: "encryptionkey@0"
I20260621 07:52:17.314850 20208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8"
format_stamp: "Formatted at 2026-06-21 07:52:17 on dist-test-slave-lbpt"
server_key: "b518ebbb4285504037dd9263f1113b44"
server_key_iv: "1517d2d40d7e1dbc00c9ffde495eb5cf"
server_key_version: "encryptionkey@0"
I20260621 07:52:17.318758 20208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:52:17.321396 20223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:17.322748 20208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:17.322888 20208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8"
format_stamp: "Formatted at 2026-06-21 07:52:17 on dist-test-slave-lbpt"
server_key: "b518ebbb4285504037dd9263f1113b44"
server_key_iv: "1517d2d40d7e1dbc00c9ffde495eb5cf"
server_key_version: "encryptionkey@0"
I20260621 07:52:17.323017 20208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:17.333824 20208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:17.337093 20208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:17.337298 20208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:17.338909 20208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:17.338970 20208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:17.339036 20208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:17.339066 20208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:52:17 dist-test-slave-lbpt krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028337, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM
I20260621 07:52:17.350176 20208 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:44809
I20260621 07:52:17.350450 20338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:44809 every 8 connection(s)
I20260621 07:52:17.351301 20208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:52:17.353709 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.339760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:38223 (local address 127.12.12.62:43865)
0621 07:52:17.339915 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:17.339920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:17.340848 (+ 928us) server_negotiation.cc:407] Connection header received
0621 07:52:17.342077 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:17.342082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:17.342150 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:17.342259 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:17.343687 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.344222 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.345594 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.345897 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.346029 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.349585 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:17.349589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:17.349592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:17.349631 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:17.351531 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:17.352197 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:17.352200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:17.352202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:17.352333 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:17.352667 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:17.352670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:17.352671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:17.352869 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:17.352952 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:17.353476 (+ 524us) server_negotiation.cc:299] Negotiation successful
0621 07:52:17.353587 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 07:52:17.354604 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.340219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43865 (local address 127.12.12.3:38223)
0621 07:52:17.340669 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:17.340722 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:17.341769 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:17.342489 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:17.342496 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:17.342901 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:17.343535 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.343550 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.344360 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.344364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.345457 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.345467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.346087 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.346091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.346268 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.347035 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:17.347055 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:17.349370 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:17.351662 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:17.351669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:17.351685 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:17.352064 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:17.352434 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:17.352437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:17.352438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:17.352575 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:17.352980 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:17.352985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:17.353221 (+ 236us) client_negotiation.cc:769] Sending connection context
0621 07:52:17.353382 (+ 161us) client_negotiation.cc:240] Negotiation successful
0621 07:52:17.353648 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":124,"threads_started":1}
I20260621 07:52:17.356060 20339 heartbeater.cc:347] Connected to a master server at 127.12.12.62:43865
I20260621 07:52:17.356215 20339 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:17.356899 20339 heartbeater.cc:511] Master 127.12.12.62:43865 requested a full tablet report, sending...
I20260621 07:52:17.358019 19855 ts_manager.cc:194] Registered new tserver with Master: 8ae8ba34f4cb4f8db4bf43eff8c00dd8 (127.12.12.3:44809)
I20260621 07:52:17.358860 19855 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:38223
I20260621 07:52:17.359413 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 20208
I20260621 07:52:17.359588 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:52:17.359908 12336 external_mini_cluster.cc:1428] Setting key 9f32c19168af7a6a1df7b849db3b116e
I20260621 07:52:17.361852 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:52:17.366461 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.358295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:60807 (local address 127.12.12.62:43865)
0621 07:52:17.358455 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:17.358460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:17.358701 (+ 241us) server_negotiation.cc:407] Connection header received
0621 07:52:17.358804 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:17.358806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:17.358844 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:17.358909 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:17.359454 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.360206 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.361305 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.361544 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.361664 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.362940 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:17.362947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:17.362953 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:17.362986 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:17.364641 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:17.365186 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:17.365189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:17.365192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:17.365289 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:17.365592 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:17.365594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:17.365596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:17.365757 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:17.365832 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:17.366151 (+ 319us) server_negotiation.cc:299] Negotiation successful
0621 07:52:17.366242 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:52:17.366585 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.358201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43865 (local address 127.12.12.3:60807)
0621 07:52:17.358613 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:17.358627 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:17.358724 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:17.358954 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:17.358957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:17.359196 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:17.359319 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.359326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.360346 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.360351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.361186 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.361196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.361690 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.361694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.361797 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.362400 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:17.362445 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:17.362796 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:17.364756 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:17.364761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:17.364764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:17.365051 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:17.365384 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:17.365390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:17.365395 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:17.365500 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:17.365859 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:17.365865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:17.365983 (+ 118us) client_negotiation.cc:769] Sending connection context
0621 07:52:17.366108 (+ 125us) client_negotiation.cc:240] Negotiation successful
0621 07:52:17.366285 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":92,"threads_started":1}
Jun 21 07:52:17 dist-test-slave-lbpt krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 07:52:17 dist-test-slave-lbpt krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:17 dist-test-slave-lbpt krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 07:52:17.387303 19905 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:17.380438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38226 (local address 127.12.12.62:43865)
0621 07:52:17.380587 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:17.380590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:17.380602 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:52:17.380644 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:17.380646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:17.380685 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:17.380789 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:17.381546 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.382082 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.383104 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.383328 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.383446 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.387122 (+ 3676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38226: BlockingRecv error: recv got EOF from 127.0.0.1:38226 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 21 07:52:17 dist-test-slave-lbpt krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM
I20260621 07:52:17.397548 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.388883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.12.12.62:43865)
0621 07:52:17.389046 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:17.389051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:17.389175 (+ 124us) server_negotiation.cc:407] Connection header received
0621 07:52:17.389329 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:17.389332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:17.389383 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:17.389450 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:17.390129 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.390752 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.391540 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.391759 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.391866 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.394273 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:17.394277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:17.394279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:17.394303 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:17.395869 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:17.396312 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:17.396316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:17.396318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:17.396398 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:17.396671 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:17.396676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:17.396678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:17.396863 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:17.397014 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:17.397259 (+ 245us) server_negotiation.cc:299] Negotiation successful
0621 07:52:17.397374 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 07:52:17.400955 19855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:52:17.404181 19855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:52:17.419718 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.413649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:40171 (local address 127.0.0.1:54042)
0621 07:52:17.414972 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:17.415012 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:17.415194 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:17.415912 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:17.415915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:17.415932 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:17.416043 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.416051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.417107 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.417111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.418368 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.418377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.419290 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.419292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.419377 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.419407 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 07:52:17.419488 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 07:52:17.419582 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"mutex_wait_us":43,"thread_start_us":84,"threads_started":1}
I20260621 07:52:17.419770 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.413838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.12.12.2:40171)
0621 07:52:17.414647 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:17.414661 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:17.415111 (+ 450us) server_negotiation.cc:407] Connection header received
0621 07:52:17.415345 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:17.415350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:17.415596 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:17.415748 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:17.416178 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.416980 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.418540 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.419182 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.419298 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.419473 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:17.419525 (+ 52us) server_negotiation.cc:299] Negotiation successful
0621 07:52:17.419622 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":121,"threads_started":1}
I20260621 07:52:17.420519 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.413649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:44809 (local address 127.0.0.1:56460)
0621 07:52:17.415487 (+ 1838us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:17.415496 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:17.415588 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:17.416067 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:17.416070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:17.416089 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:17.416185 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.416191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.417534 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.417536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.419038 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.419048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.420130 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.420133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.420219 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.420242 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 07:52:17.420311 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 07:52:17.420383 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1783,"mutex_wait_us":1147,"thread_start_us":88,"threads_started":1}
I20260621 07:52:17.420571 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.413916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56460 (local address 127.12.12.3:44809)
0621 07:52:17.414648 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:17.414661 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:17.415582 (+ 921us) server_negotiation.cc:407] Connection header received
0621 07:52:17.415701 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:17.415705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:17.415913 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:17.416083 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:17.416278 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.417418 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.419185 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.420003 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.420131 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.420283 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:17.420337 (+ 54us) server_negotiation.cc:299] Negotiation successful
0621 07:52:17.420437 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":144,"threads_started":1}
I20260621 07:52:17.422600 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.413956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:42795 (local address 127.0.0.1:40192)
0621 07:52:17.415104 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:17.415117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:17.415304 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:17.415794 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:17.415800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:17.415822 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:17.415992 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.416003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.417524 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.417527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.419038 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.419048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.422152 (+ 3104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.422155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.422257 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.422279 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:52:17.422355 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 07:52:17.422456 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"mutex_wait_us":133,"thread_start_us":97,"threads_started":1}
I20260621 07:52:17.422642 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.414401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.12.12.1:42795)
0621 07:52:17.414812 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:17.414819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:17.415217 (+ 398us) server_negotiation.cc:407] Connection header received
0621 07:52:17.415436 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:17.415441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:17.415595 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:17.415766 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:17.416131 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.417345 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.421403 (+ 4058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.422019 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.422155 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.422297 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:17.422393 (+ 96us) server_negotiation.cc:299] Negotiation successful
0621 07:52:17.422510 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"spinlock_wait_cycles":6016,"thread_start_us":81,"threads_started":1}
I20260621 07:52:17.423123 20273 tablet_service.cc:1467] Processing CreateTablet for tablet c3d20bba1d56448e839690aace5a380c (DEFAULT_TABLE table=test-table [id=e469179ad59b4121b8a3b0e1b0dff787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:17.423024 20133 tablet_service.cc:1467] Processing CreateTablet for tablet c3d20bba1d56448e839690aace5a380c (DEFAULT_TABLE table=test-table [id=e469179ad59b4121b8a3b0e1b0dff787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:17.424218 20133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d20bba1d56448e839690aace5a380c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:17.424218 20273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d20bba1d56448e839690aace5a380c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:17.432756 20367 tablet_bootstrap.cc:492] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Bootstrap starting.
I20260621 07:52:17.432485 19993 tablet_service.cc:1467] Processing CreateTablet for tablet c3d20bba1d56448e839690aace5a380c (DEFAULT_TABLE table=test-table [id=e469179ad59b4121b8a3b0e1b0dff787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:17.433279 20368 tablet_bootstrap.cc:492] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342: Bootstrap starting.
I20260621 07:52:17.433732 19993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d20bba1d56448e839690aace5a380c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:17.435699 20368 tablet_bootstrap.cc:654] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:17.436569 20367 tablet_bootstrap.cc:654] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:17.436558 20368 log.cc:824] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:17.437424 20367 log.cc:824] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:17.438707 20368 tablet_bootstrap.cc:492] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342: No bootstrap required, opened a new log
I20260621 07:52:17.438908 20368 ts_tablet_manager.cc:1397] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:52:17.439261 20367 tablet_bootstrap.cc:492] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: No bootstrap required, opened a new log
I20260621 07:52:17.439463 20367 ts_tablet_manager.cc:1397] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260621 07:52:17.440966 20371 tablet_bootstrap.cc:492] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165: Bootstrap starting.
I20260621 07:52:17.442680 20368 raft_consensus.cc:348] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } }
I20260621 07:52:17.442967 20368 raft_consensus.cc:374] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:17.443028 20368 raft_consensus.cc:729] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4777f89fad4de0aac931ffd1538342, State: Initialized, Role: FOLLOWER
I20260621 07:52:17.442991 20367 raft_consensus.cc:348] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } }
I20260621 07:52:17.443246 20367 raft_consensus.cc:374] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:17.443300 20367 raft_consensus.cc:729] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae8ba34f4cb4f8db4bf43eff8c00dd8, State: Initialized, Role: FOLLOWER
I20260621 07:52:17.443480 20371 tablet_bootstrap.cc:654] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:17.443562 20368 consensus_queue.cc:260] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } }
I20260621 07:52:17.443759 20367 consensus_queue.cc:260] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } }
I20260621 07:52:17.444315 20371 log.cc:824] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:17.444414 20339 heartbeater.cc:503] Master 127.12.12.62:43865 was elected leader, sending a full tablet report...
I20260621 07:52:17.444654 20368 ts_tablet_manager.cc:1428] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:52:17.445011 20199 heartbeater.cc:503] Master 127.12.12.62:43865 was elected leader, sending a full tablet report...
I20260621 07:52:17.445415 20367 ts_tablet_manager.cc:1428] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260621 07:52:17.447177 20371 tablet_bootstrap.cc:492] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165: No bootstrap required, opened a new log
I20260621 07:52:17.447381 20371 ts_tablet_manager.cc:1397] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260621 07:52:17.451354 20371 raft_consensus.cc:348] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } }
I20260621 07:52:17.451629 20371 raft_consensus.cc:374] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:17.451715 20371 raft_consensus.cc:729] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da12b3e95de94515857e94d262a8b165, State: Initialized, Role: FOLLOWER
I20260621 07:52:17.452137 20371 consensus_queue.cc:260] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } }
I20260621 07:52:17.452749 20059 heartbeater.cc:503] Master 127.12.12.62:43865 was elected leader, sending a full tablet report...
I20260621 07:52:17.453073 20371 ts_tablet_manager.cc:1428] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260621 07:52:17.602458 20340 tablet.cc:2369] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:52:17.628875 20200 tablet.cc:2369] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:52:17.640031 20060 tablet.cc:2369] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:52:17.664650 20373 raft_consensus.cc:482] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:52:17.664886 20373 raft_consensus.cc:504] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } }
I20260621 07:52:17.666229 20373 leader_election.cc:288] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de4777f89fad4de0aac931ffd1538342 (127.12.12.2:40171), da12b3e95de94515857e94d262a8b165 (127.12.12.1:42795)
I20260621 07:52:17.670704 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.666508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:40171 (local address 127.12.12.3:53167)
0621 07:52:17.666731 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:17.666747 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:17.666864 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:17.667224 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:17.667227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:17.667242 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:17.667389 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.667395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.668199 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.668202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.669372 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.669383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.670145 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.670148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.670228 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.670247 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 07:52:17.670404 (+ 157us) client_negotiation.cc:240] Negotiation successful
0621 07:52:17.670521 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":23}
I20260621 07:52:17.670701 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.666612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:53167 (local address 127.12.12.2:40171)
0621 07:52:17.666781 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:17.666814 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:17.666834 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:52:17.667026 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:17.667029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:17.667106 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:17.667188 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:17.667514 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.668090 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.669515 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.670019 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.670135 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.670204 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:17.670475 (+ 271us) server_negotiation.cc:299] Negotiation successful
0621 07:52:17.670564 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:52:17.671116 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.666720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:34159 (local address 127.12.12.1:42795)
0621 07:52:17.666843 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:17.666848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:17.667101 (+ 253us) server_negotiation.cc:407] Connection header received
0621 07:52:17.667222 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:17.667225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:17.667278 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:17.667354 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:17.668016 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.668584 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.669843 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.670528 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.670635 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.670692 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:17.670943 (+ 251us) server_negotiation.cc:299] Negotiation successful
0621 07:52:17.671025 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 07:52:17.671469 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.666606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:42795 (local address 127.12.12.3:34159)
0621 07:52:17.667009 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:17.667020 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:17.667115 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:17.667387 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:17.667390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:17.667402 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:17.667894 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.667901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.668704 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.668707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.669727 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.669734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.670644 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.670647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.670753 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.670779 (+ 26us) client_negotiation.cc:769] Sending connection context
0621 07:52:17.671273 (+ 494us) client_negotiation.cc:240] Negotiation successful
0621 07:52:17.671351 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":1,"thread_start_us":131,"threads_started":1}
I20260621 07:52:17.671679 20153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d20bba1d56448e839690aace5a380c" candidate_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de4777f89fad4de0aac931ffd1538342" is_pre_election: true
I20260621 07:52:17.672010 20153 raft_consensus.cc:2454] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae8ba34f4cb4f8db4bf43eff8c00dd8 in term 0.
I20260621 07:52:17.672051 20013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d20bba1d56448e839690aace5a380c" candidate_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da12b3e95de94515857e94d262a8b165" is_pre_election: true
I20260621 07:52:17.672360 20013 raft_consensus.cc:2454] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae8ba34f4cb4f8db4bf43eff8c00dd8 in term 0.
I20260621 07:52:17.672536 20224 leader_election.cc:302] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae8ba34f4cb4f8db4bf43eff8c00dd8, de4777f89fad4de0aac931ffd1538342; no voters:
I20260621 07:52:17.672821 20373 raft_consensus.cc:2781] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:52:17.672916 20373 raft_consensus.cc:482] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:52:17.672960 20373 raft_consensus.cc:3037] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:17.673971 20373 raft_consensus.cc:504] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } }
I20260621 07:52:17.674324 20373 leader_election.cc:288] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [CANDIDATE]: Term 1 election: Requested vote from peers de4777f89fad4de0aac931ffd1538342 (127.12.12.2:40171), da12b3e95de94515857e94d262a8b165 (127.12.12.1:42795)
I20260621 07:52:17.674778 20153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d20bba1d56448e839690aace5a380c" candidate_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de4777f89fad4de0aac931ffd1538342"
I20260621 07:52:17.674800 20013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d20bba1d56448e839690aace5a380c" candidate_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da12b3e95de94515857e94d262a8b165"
I20260621 07:52:17.674886 20153 raft_consensus.cc:3037] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:17.674971 20013 raft_consensus.cc:3037] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:17.675957 20153 raft_consensus.cc:2454] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae8ba34f4cb4f8db4bf43eff8c00dd8 in term 1.
I20260621 07:52:17.676074 20013 raft_consensus.cc:2454] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae8ba34f4cb4f8db4bf43eff8c00dd8 in term 1.
I20260621 07:52:17.676272 20224 leader_election.cc:302] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae8ba34f4cb4f8db4bf43eff8c00dd8, de4777f89fad4de0aac931ffd1538342; no voters:
I20260621 07:52:17.676455 20373 raft_consensus.cc:2781] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:17.676666 20373 raft_consensus.cc:686] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 LEADER]: Becoming Leader. State: Replica: 8ae8ba34f4cb4f8db4bf43eff8c00dd8, State: Running, Role: LEADER
I20260621 07:52:17.676993 20373 consensus_queue.cc:237] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } }
I20260621 07:52:17.680344 19855 catalog_manager.cc:5495] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ae8ba34f4cb4f8db4bf43eff8c00dd8 (127.12.12.3). New cstate: current_term: 1 leader_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } health_report { overall_health: UNKNOWN } } }
I20260621 07:52:17.713249 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.709672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.12.12.3:44809)
0621 07:52:17.709799 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:17.709803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:17.709815 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:52:17.709893 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:17.709895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:17.709948 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:17.710038 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:52:17.710247 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.710867 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.711758 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.711976 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.712053 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.712323 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:52:17.712815 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:52:17.712910 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:17.713053 (+ 143us) server_negotiation.cc:299] Negotiation successful
0621 07:52:17.713106 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:52:17.719664 20013 raft_consensus.cc:1261] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165 [term 1 FOLLOWER]: Refusing update from remote peer 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:17.719663 20153 raft_consensus.cc:1261] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Refusing update from remote peer 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:17.720443 20377 consensus_queue.cc:1035] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:17.720629 20373 consensus_queue.cc:1035] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:17.729336 20380 mvcc.cc:205] Tried to move back new op lower bound from 7299188071291400192 to 7299188071127539712. Current Snapshot: MvccSnapshot[applied={T|T < 7299188071291400192 or (T in {7299188071291400192})}]
I20260621 07:52:17.729553 20382 mvcc.cc:205] Tried to move back new op lower bound from 7299188071291400192 to 7299188071127539712. Current Snapshot: MvccSnapshot[applied={T|T < 7299188071291400192}]
I20260621 07:52:17.736662 19855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:17.736874 19855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:17.739609 19855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c: Sending DeleteTablet for 3 replicas of tablet c3d20bba1d56448e839690aace5a380c
I20260621 07:52:17.740500 20133 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d20bba1d56448e839690aace5a380c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:17 UTC) from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:54042
I20260621 07:52:17.740470 20273 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d20bba1d56448e839690aace5a380c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:17 UTC) from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:56460
I20260621 07:52:17.740501 19993 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d20bba1d56448e839690aace5a380c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:17 UTC) from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:40192
I20260621 07:52:17.741142 20389 tablet_replica.cc:330] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342: stopping tablet replica
I20260621 07:52:17.741499 20389 raft_consensus.cc:2229] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:17.741852 20389 raft_consensus.cc:2258] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:17.742076 19855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 07:52:17.742229 20390 tablet_replica.cc:330] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: stopping tablet replica
I20260621 07:52:17.742530 20390 raft_consensus.cc:2229] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 LEADER]: Raft consensus shutting down.
W20260621 07:52:17.742798 19855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:52:17.742887 20390 raft_consensus.cc:2258] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:17.743257 20391 tablet_replica.cc:330] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165: stopping tablet replica
I20260621 07:52:17.743880 20389 ts_tablet_manager.cc:1905] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:17.744028 20391 raft_consensus.cc:2229] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:17.744341 20391 raft_consensus.cc:2258] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:17.747077 20389 ts_tablet_manager.cc:1918] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:17.747231 20389 log.cc:1196] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/c3d20bba1d56448e839690aace5a380c
I20260621 07:52:17.747510 20390 ts_tablet_manager.cc:1905] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:17.747666 20389 ts_tablet_manager.cc:1939] T c3d20bba1d56448e839690aace5a380c P de4777f89fad4de0aac931ffd1538342: Deleting consensus metadata
I20260621 07:52:17.748093 20133 tablet_service.cc:1467] Processing CreateTablet for tablet 146b24241e5946898f367fb6e13ae39c (DEFAULT_TABLE table=test-table [id=dc5428b61f654c7fbcd8b973726b7913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:17.748219 19993 tablet_service.cc:1467] Processing CreateTablet for tablet 146b24241e5946898f367fb6e13ae39c (DEFAULT_TABLE table=test-table [id=dc5428b61f654c7fbcd8b973726b7913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:17.748363 20133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 146b24241e5946898f367fb6e13ae39c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:17.748464 19993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 146b24241e5946898f367fb6e13ae39c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:17.750624 20368 tablet_bootstrap.cc:492] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342: Bootstrap starting.
I20260621 07:52:17.750743 19840 catalog_manager.cc:4841] TS de4777f89fad4de0aac931ffd1538342 (127.12.12.2:40171): tablet c3d20bba1d56448e839690aace5a380c (table test-table [id=e469179ad59b4121b8a3b0e1b0dff787]) successfully deleted
I20260621 07:52:17.751447 20273 tablet_service.cc:1467] Processing CreateTablet for tablet 146b24241e5946898f367fb6e13ae39c (DEFAULT_TABLE table=test-table [id=dc5428b61f654c7fbcd8b973726b7913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:17.751533 20368 tablet_bootstrap.cc:654] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:17.751688 20273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 146b24241e5946898f367fb6e13ae39c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:17.751672 20391 ts_tablet_manager.cc:1905] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:17.753935 20391 ts_tablet_manager.cc:1918] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:17.754038 20391 log.cc:1196] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/wal/wals/c3d20bba1d56448e839690aace5a380c
I20260621 07:52:17.754099 20367 tablet_bootstrap.cc:492] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Bootstrap starting.
I20260621 07:52:17.754374 20391 ts_tablet_manager.cc:1939] T c3d20bba1d56448e839690aace5a380c P da12b3e95de94515857e94d262a8b165: Deleting consensus metadata
I20260621 07:52:17.754681 20371 tablet_bootstrap.cc:492] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165: Bootstrap starting.
I20260621 07:52:17.755055 20367 tablet_bootstrap.cc:654] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:17.755728 20371 tablet_bootstrap.cc:654] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:17.755754 20368 tablet_bootstrap.cc:492] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342: No bootstrap required, opened a new log
I20260621 07:52:17.755859 20368 ts_tablet_manager.cc:1397] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260621 07:52:17.756356 20368 raft_consensus.cc:348] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:17.756487 20368 raft_consensus.cc:374] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:17.756526 20368 raft_consensus.cc:729] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4777f89fad4de0aac931ffd1538342, State: Initialized, Role: FOLLOWER
I20260621 07:52:17.756685 20368 consensus_queue.cc:260] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:17.756735 20390 ts_tablet_manager.cc:1918] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:17.756872 20390 log.cc:1196] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/c3d20bba1d56448e839690aace5a380c
I20260621 07:52:17.757213 20390 ts_tablet_manager.cc:1939] T c3d20bba1d56448e839690aace5a380c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Deleting consensus metadata
I20260621 07:52:17.757529 20371 tablet_bootstrap.cc:492] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165: No bootstrap required, opened a new log
I20260621 07:52:17.757637 20371 ts_tablet_manager.cc:1397] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 07:52:17.758105 20371 raft_consensus.cc:348] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:17.758221 20371 raft_consensus.cc:374] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:17.758263 20371 raft_consensus.cc:729] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da12b3e95de94515857e94d262a8b165, State: Initialized, Role: FOLLOWER
I20260621 07:52:17.758488 19842 catalog_manager.cc:4841] TS 8ae8ba34f4cb4f8db4bf43eff8c00dd8 (127.12.12.3:44809): tablet c3d20bba1d56448e839690aace5a380c (table test-table [id=e469179ad59b4121b8a3b0e1b0dff787]) successfully deleted
I20260621 07:52:17.758481 20371 consensus_queue.cc:260] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:17.759115 20368 ts_tablet_manager.cc:1428] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260621 07:52:17.759148 20371 ts_tablet_manager.cc:1428] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:52:17.759526 19843 catalog_manager.cc:4841] TS da12b3e95de94515857e94d262a8b165 (127.12.12.1:42795): tablet c3d20bba1d56448e839690aace5a380c (table test-table [id=e469179ad59b4121b8a3b0e1b0dff787]) successfully deleted
I20260621 07:52:17.760176 20367 tablet_bootstrap.cc:492] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: No bootstrap required, opened a new log
I20260621 07:52:17.760257 20367 ts_tablet_manager.cc:1397] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260621 07:52:17.760679 20367 raft_consensus.cc:348] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:17.760778 20367 raft_consensus.cc:374] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:17.760807 20367 raft_consensus.cc:729] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae8ba34f4cb4f8db4bf43eff8c00dd8, State: Initialized, Role: FOLLOWER
I20260621 07:52:17.760928 20367 consensus_queue.cc:260] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:17.761152 20367 ts_tablet_manager.cc:1428] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:52:17.874454 20375 raft_consensus.cc:482] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:52:17.874630 20375 raft_consensus.cc:504] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:17.876075 20375 leader_election.cc:288] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de4777f89fad4de0aac931ffd1538342 (127.12.12.2:40171), 8ae8ba34f4cb4f8db4bf43eff8c00dd8 (127.12.12.3:44809)
I20260621 07:52:17.880956 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.876520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:51613 (local address 127.12.12.2:40171)
0621 07:52:17.876733 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:17.876737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:17.876983 (+ 246us) server_negotiation.cc:407] Connection header received
0621 07:52:17.877080 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:17.877083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:17.877135 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:17.877235 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:17.877555 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.878325 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.879930 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.880348 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.880478 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.880551 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:17.880739 (+ 188us) server_negotiation.cc:299] Negotiation successful
0621 07:52:17.880829 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260621 07:52:17.880990 20396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.876338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:40171 (local address 127.12.12.1:51613)
0621 07:52:17.876881 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:17.876894 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:17.876986 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:17.877269 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:17.877275 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:17.877299 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:17.877431 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.877439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.878461 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.878464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.879761 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.879773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.880499 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.880503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.880618 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.880643 (+ 25us) client_negotiation.cc:769] Sending connection context
0621 07:52:17.880716 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 07:52:17.880804 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":307,"thread_start_us":84,"threads_started":1}
I20260621 07:52:17.881062 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.876451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:48595 (local address 127.12.12.3:44809)
0621 07:52:17.876603 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:17.876608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:17.876811 (+ 203us) server_negotiation.cc:407] Connection header received
0621 07:52:17.877053 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:17.877057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:17.877109 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:17.877159 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:17.877593 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.878099 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.879909 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.880514 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.880609 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.880669 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:17.880854 (+ 185us) server_negotiation.cc:299] Negotiation successful
0621 07:52:17.880931 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:52:17.881474 20153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146b24241e5946898f367fb6e13ae39c" candidate_uuid: "da12b3e95de94515857e94d262a8b165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de4777f89fad4de0aac931ffd1538342" is_pre_election: true
I20260621 07:52:17.881633 20153 raft_consensus.cc:2454] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da12b3e95de94515857e94d262a8b165 in term 0.
I20260621 07:52:17.881672 20395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.876338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:44809 (local address 127.12.12.1:48595)
0621 07:52:17.876696 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:17.876711 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:17.876906 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:17.877345 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:17.877348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:17.877360 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:17.877455 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.877462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.878217 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.878223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.879761 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.879773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.880619 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.880622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.880721 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.880739 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 07:52:17.881190 (+ 451us) client_negotiation.cc:240] Negotiation successful
0621 07:52:17.881272 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":63,"thread_start_us":74,"threads_started":1}
I20260621 07:52:17.882030 19944 leader_election.cc:302] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da12b3e95de94515857e94d262a8b165, de4777f89fad4de0aac931ffd1538342; no voters:
I20260621 07:52:17.882233 20293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146b24241e5946898f367fb6e13ae39c" candidate_uuid: "da12b3e95de94515857e94d262a8b165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" is_pre_election: true
I20260621 07:52:17.882499 20375 raft_consensus.cc:2781] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:52:17.882507 20293 raft_consensus.cc:2454] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da12b3e95de94515857e94d262a8b165 in term 0.
I20260621 07:52:17.882556 20375 raft_consensus.cc:482] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:52:17.882594 20375 raft_consensus.cc:3037] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:17.883831 20375 raft_consensus.cc:504] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:17.884268 20375 leader_election.cc:288] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [CANDIDATE]: Term 1 election: Requested vote from peers de4777f89fad4de0aac931ffd1538342 (127.12.12.2:40171), 8ae8ba34f4cb4f8db4bf43eff8c00dd8 (127.12.12.3:44809)
I20260621 07:52:17.884672 20153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146b24241e5946898f367fb6e13ae39c" candidate_uuid: "da12b3e95de94515857e94d262a8b165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de4777f89fad4de0aac931ffd1538342"
I20260621 07:52:17.884706 20293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146b24241e5946898f367fb6e13ae39c" candidate_uuid: "da12b3e95de94515857e94d262a8b165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8"
I20260621 07:52:17.884824 20293 raft_consensus.cc:3037] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:17.884817 20153 raft_consensus.cc:3037] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:17.885985 20153 raft_consensus.cc:2454] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da12b3e95de94515857e94d262a8b165 in term 1.
I20260621 07:52:17.886040 20293 raft_consensus.cc:2454] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da12b3e95de94515857e94d262a8b165 in term 1.
I20260621 07:52:17.886309 19944 leader_election.cc:302] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da12b3e95de94515857e94d262a8b165, de4777f89fad4de0aac931ffd1538342; no voters:
I20260621 07:52:17.886550 20375 raft_consensus.cc:2781] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:17.886797 20375 raft_consensus.cc:686] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [term 1 LEADER]: Becoming Leader. State: Replica: da12b3e95de94515857e94d262a8b165, State: Running, Role: LEADER
I20260621 07:52:17.887190 20375 consensus_queue.cc:237] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:17.891263 19854 catalog_manager.cc:5495] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 reported cstate change: term changed from 0 to 1, leader changed from <none> to da12b3e95de94515857e94d262a8b165 (127.12.12.1). New cstate: current_term: 1 leader_uuid: "da12b3e95de94515857e94d262a8b165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } health_report { overall_health: UNKNOWN } } }
Jun 21 07:52:17 dist-test-slave-lbpt krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028336, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.62@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM
I20260621 07:52:17.940542 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.932149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43865 (local address 127.0.0.1:38240)
0621 07:52:17.932410 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:17.932430 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:17.932544 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:17.932825 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:17.932828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:17.933241 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:17.933346 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.933353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.934123 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.934126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.934748 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:17.934757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.935226 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.935229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:17.935298 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.935855 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:17.935866 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:17.937283 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:17.939019 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:17.939028 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:17.939033 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:17.939276 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:17.939571 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:17.939573 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:17.939575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:17.939657 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:17.940022 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:17.940027 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:17.940128 (+ 101us) client_negotiation.cc:769] Sending connection context
0621 07:52:17.940222 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 07:52:17.940349 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
I20260621 07:52:17.940544 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:17.932286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.12.12.62:43865)
0621 07:52:17.932501 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:17.932505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:17.932519 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:52:17.932631 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:17.932633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:17.932671 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:17.932751 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:17.933463 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.933993 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.934879 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:17.935096 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:17.935211 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:17.937472 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:17.937476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:17.937478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:17.937498 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:17.938879 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:17.939373 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:17.939376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:17.939377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:17.939465 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:17.939740 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:17.939742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:17.939744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:17.939928 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:17.940005 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:17.940277 (+ 272us) server_negotiation.cc:299] Negotiation successful
0621 07:52:17.940411 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
I20260621 07:52:17.944547 19854 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:38240:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 07:52:17.945308 19854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:52:17.950343 19993 tablet_service.cc:1467] Processing CreateTablet for tablet c0334aa96dc744bca8244bf4f238ea5c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e10f0a6eea74c2a93d606dd8c2379a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:52:17.950342 20133 tablet_service.cc:1467] Processing CreateTablet for tablet c0334aa96dc744bca8244bf4f238ea5c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e10f0a6eea74c2a93d606dd8c2379a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:52:17.950381 20273 tablet_service.cc:1467] Processing CreateTablet for tablet c0334aa96dc744bca8244bf4f238ea5c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e10f0a6eea74c2a93d606dd8c2379a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:52:17.950790 19993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0334aa96dc744bca8244bf4f238ea5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:17.950790 20133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0334aa96dc744bca8244bf4f238ea5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:17.950853 20273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0334aa96dc744bca8244bf4f238ea5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:17.953092 20367 tablet_bootstrap.cc:492] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Bootstrap starting.
I20260621 07:52:17.953647 20371 tablet_bootstrap.cc:492] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165: Bootstrap starting.
I20260621 07:52:17.953647 20368 tablet_bootstrap.cc:492] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342: Bootstrap starting.
I20260621 07:52:17.954012 20367 tablet_bootstrap.cc:654] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:17.954556 20371 tablet_bootstrap.cc:654] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:17.954644 20368 tablet_bootstrap.cc:654] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:17.955691 20367 tablet_bootstrap.cc:492] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: No bootstrap required, opened a new log
I20260621 07:52:17.955807 20367 ts_tablet_manager.cc:1397] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 07:52:17.955849 20368 tablet_bootstrap.cc:492] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342: No bootstrap required, opened a new log
I20260621 07:52:17.955920 20371 tablet_bootstrap.cc:492] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165: No bootstrap required, opened a new log
I20260621 07:52:17.955937 20368 ts_tablet_manager.cc:1397] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:52:17.956007 20371 ts_tablet_manager.cc:1397] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260621 07:52:17.956296 20367 raft_consensus.cc:348] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:17.956426 20367 raft_consensus.cc:374] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:17.956403 20368 raft_consensus.cc:348] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:17.956467 20367 raft_consensus.cc:729] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae8ba34f4cb4f8db4bf43eff8c00dd8, State: Initialized, Role: FOLLOWER
I20260621 07:52:17.956444 20371 raft_consensus.cc:348] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:17.956526 20368 raft_consensus.cc:374] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:17.956540 20371 raft_consensus.cc:374] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:17.956573 20368 raft_consensus.cc:729] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4777f89fad4de0aac931ffd1538342, State: Initialized, Role: FOLLOWER
I20260621 07:52:17.956578 20371 raft_consensus.cc:729] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da12b3e95de94515857e94d262a8b165, State: Initialized, Role: FOLLOWER
I20260621 07:52:17.956619 20367 consensus_queue.cc:260] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:17.956700 20368 consensus_queue.cc:260] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:17.956699 20371 consensus_queue.cc:260] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:17.956979 20368 ts_tablet_manager.cc:1428] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:52:17.956964 20371 ts_tablet_manager.cc:1428] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260621 07:52:17.957022 20373 tablet_replica.cc:439] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } }
I20260621 07:52:17.957155 20373 tablet_replica.cc:442] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:52:17.957182 20372 tablet_replica.cc:439] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } }
I20260621 07:52:17.957298 20372 tablet_replica.cc:442] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:52:17.957799 20367 ts_tablet_manager.cc:1428] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:52:17.958168 20375 tablet_replica.cc:439] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } }
I20260621 07:52:17.958321 20375 tablet_replica.cc:442] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:52:18.042738 20372 raft_consensus.cc:482] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:52:18.042886 20372 raft_consensus.cc:504] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.044021 20372 leader_election.cc:288] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da12b3e95de94515857e94d262a8b165 (127.12.12.1:42795), 8ae8ba34f4cb4f8db4bf43eff8c00dd8 (127.12.12.3:44809)
I20260621 07:52:18.048163 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:18.044749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:39551 (local address 127.12.12.3:44809)
0621 07:52:18.044894 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:18.044898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:18.044911 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:52:18.044959 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:18.044961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:18.045003 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:18.045105 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:18.045461 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.046023 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.047157 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.047557 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.047680 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:18.047746 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:18.047946 (+ 200us) server_negotiation.cc:299] Negotiation successful
0621 07:52:18.048020 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:52:18.048188 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:18.044372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:39971 (local address 127.12.12.1:42795)
0621 07:52:18.044526 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:18.044532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:18.044640 (+ 108us) server_negotiation.cc:407] Connection header received
0621 07:52:18.044919 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:18.044922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:18.044962 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:18.045044 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:18.045336 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.045843 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.047119 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.047555 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.047678 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:18.047751 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:18.047977 (+ 226us) server_negotiation.cc:299] Negotiation successful
0621 07:52:18.048060 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:52:18.048223 20409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:18.044206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:42795 (local address 127.12.12.2:39971)
0621 07:52:18.044541 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:18.044558 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:18.044751 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:18.045067 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:18.045070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:18.045087 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:18.045227 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:18.045233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.045952 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.045958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:18.046993 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:18.047003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.047681 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.047685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:18.047854 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:18.047872 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 07:52:18.047962 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 07:52:18.048048 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":48,"spinlock_wait_cycles":18688,"thread_start_us":154,"threads_started":1}
I20260621 07:52:18.048228 20410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:18.044370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:44809 (local address 127.12.12.2:39551)
0621 07:52:18.044622 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:18.044635 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:18.044751 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:18.045116 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:18.045119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:18.045137 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:18.045293 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:18.045342 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.046132 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.046138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:18.047029 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:18.047039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.047682 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.047686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:18.047810 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:18.047839 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 07:52:18.047928 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 07:52:18.048012 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":31,"spinlock_wait_cycles":27904,"thread_start_us":92,"threads_started":1}
I20260621 07:52:18.048796 20013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0334aa96dc744bca8244bf4f238ea5c" candidate_uuid: "de4777f89fad4de0aac931ffd1538342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da12b3e95de94515857e94d262a8b165" is_pre_election: true
I20260621 07:52:18.048905 20293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0334aa96dc744bca8244bf4f238ea5c" candidate_uuid: "de4777f89fad4de0aac931ffd1538342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" is_pre_election: true
I20260621 07:52:18.048992 20013 raft_consensus.cc:2454] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de4777f89fad4de0aac931ffd1538342 in term 0.
I20260621 07:52:18.049048 20293 raft_consensus.cc:2454] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de4777f89fad4de0aac931ffd1538342 in term 0.
I20260621 07:52:18.049425 20086 leader_election.cc:302] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8ae8ba34f4cb4f8db4bf43eff8c00dd8, da12b3e95de94515857e94d262a8b165, de4777f89fad4de0aac931ffd1538342; no voters:
I20260621 07:52:18.049726 20372 raft_consensus.cc:2781] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:52:18.049784 20372 raft_consensus.cc:482] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:52:18.049818 20372 raft_consensus.cc:3037] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:18.050879 20372 raft_consensus.cc:504] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.051301 20372 leader_election.cc:288] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [CANDIDATE]: Term 1 election: Requested vote from peers da12b3e95de94515857e94d262a8b165 (127.12.12.1:42795), 8ae8ba34f4cb4f8db4bf43eff8c00dd8 (127.12.12.3:44809)
I20260621 07:52:18.051663 20293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0334aa96dc744bca8244bf4f238ea5c" candidate_uuid: "de4777f89fad4de0aac931ffd1538342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8"
I20260621 07:52:18.051664 20013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0334aa96dc744bca8244bf4f238ea5c" candidate_uuid: "de4777f89fad4de0aac931ffd1538342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da12b3e95de94515857e94d262a8b165"
I20260621 07:52:18.051781 20293 raft_consensus.cc:3037] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:18.051795 20013 raft_consensus.cc:3037] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:18.053059 20013 raft_consensus.cc:2454] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de4777f89fad4de0aac931ffd1538342 in term 1.
I20260621 07:52:18.053184 20293 raft_consensus.cc:2454] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de4777f89fad4de0aac931ffd1538342 in term 1.
I20260621 07:52:18.053428 20087 leader_election.cc:302] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da12b3e95de94515857e94d262a8b165, de4777f89fad4de0aac931ffd1538342; no voters:
I20260621 07:52:18.053619 20372 raft_consensus.cc:2781] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:18.053838 20372 raft_consensus.cc:686] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [term 1 LEADER]: Becoming Leader. State: Replica: de4777f89fad4de0aac931ffd1538342, State: Running, Role: LEADER
I20260621 07:52:18.054103 20372 consensus_queue.cc:237] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.055485 20411 tablet_replica.cc:439] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342: TxnStatusTablet state changed. Reason: New leader de4777f89fad4de0aac931ffd1538342. Latest consensus state: current_term: 1 leader_uuid: "de4777f89fad4de0aac931ffd1538342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } }
I20260621 07:52:18.055611 20411 tablet_replica.cc:442] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:52:18.055977 20413 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 07:52:18.056136 20413 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 07:52:18.057054 19854 catalog_manager.cc:5495] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 reported cstate change: term changed from 0 to 1, leader changed from <none> to de4777f89fad4de0aac931ffd1538342 (127.12.12.2). New cstate: current_term: 1 leader_uuid: "de4777f89fad4de0aac931ffd1538342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } health_report { overall_health: UNKNOWN } } }
I20260621 07:52:18.086748 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:18.082369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:40171 (local address 127.0.0.1:54056)
0621 07:52:18.082683 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:18.082698 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:18.082815 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:18.083143 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:18.083146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:18.083155 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:52:18.083266 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:18.083272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.084018 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.084022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:18.084989 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:18.084999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.085453 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.085457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:18.085554 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:18.085609 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:52:18.086374 (+ 765us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:52:18.086391 (+ 17us) client_negotiation.cc:769] Sending connection context
0621 07:52:18.086489 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 07:52:18.086583 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260621 07:52:18.086769 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:18.082514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54056 (local address 127.12.12.2:40171)
0621 07:52:18.082641 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:18.082645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:18.082785 (+ 140us) server_negotiation.cc:407] Connection header received
0621 07:52:18.082961 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:18.082964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:18.083020 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:18.083110 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:52:18.083369 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.083867 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.085114 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.085342 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.085425 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:18.085806 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:52:18.086267 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:52:18.086368 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:18.086529 (+ 161us) server_negotiation.cc:299] Negotiation successful
0621 07:52:18.086599 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:52:18.089954 20013 raft_consensus.cc:1261] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165 [term 1 FOLLOWER]: Refusing update from remote peer de4777f89fad4de0aac931ffd1538342: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:18.090144 20293 raft_consensus.cc:1261] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Refusing update from remote peer de4777f89fad4de0aac931ffd1538342: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:18.090510 20372 consensus_queue.cc:1035] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:18.090751 20411 consensus_queue.cc:1035] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:18.092317 20373 tablet_replica.cc:439] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: TxnStatusTablet state changed. Reason: New leader de4777f89fad4de0aac931ffd1538342. Latest consensus state: current_term: 1 leader_uuid: "de4777f89fad4de0aac931ffd1538342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } }
I20260621 07:52:18.092293 20375 tablet_replica.cc:439] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165: TxnStatusTablet state changed. Reason: New leader de4777f89fad4de0aac931ffd1538342. Latest consensus state: current_term: 1 leader_uuid: "de4777f89fad4de0aac931ffd1538342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } }
I20260621 07:52:18.092460 20373 tablet_replica.cc:442] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:52:18.092461 20375 tablet_replica.cc:442] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:52:18.094717 20372 tablet_replica.cc:439] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de4777f89fad4de0aac931ffd1538342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } }
I20260621 07:52:18.094833 20372 tablet_replica.cc:442] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:52:18.094936 20381 mvcc.cc:205] Tried to move back new op lower bound from 7299188072810774528 to 7299188072672301056. Current Snapshot: MvccSnapshot[applied={T|T < 7299188072810774528}]
I20260621 07:52:18.094970 20375 tablet_replica.cc:439] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de4777f89fad4de0aac931ffd1538342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } }
I20260621 07:52:18.095088 20375 tablet_replica.cc:442] T c0334aa96dc744bca8244bf4f238ea5c P da12b3e95de94515857e94d262a8b165: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:52:18.098090 20411 tablet_replica.cc:439] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de4777f89fad4de0aac931ffd1538342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } }
I20260621 07:52:18.098222 20411 tablet_replica.cc:442] T c0334aa96dc744bca8244bf4f238ea5c P de4777f89fad4de0aac931ffd1538342: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:52:18.099308 20373 tablet_replica.cc:439] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de4777f89fad4de0aac931ffd1538342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } } }
I20260621 07:52:18.099431 20373 tablet_replica.cc:442] T c0334aa96dc744bca8244bf4f238ea5c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:52:18.104631 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:18.101050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40204 (local address 127.12.12.1:42795)
0621 07:52:18.101153 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:18.101158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:18.101180 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:52:18.101248 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:18.101250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:18.101302 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:18.101404 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:52:18.101672 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.102208 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.103149 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.103393 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.103505 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:18.103773 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:52:18.104221 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:52:18.104354 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:18.104421 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 07:52:18.104490 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 07:52:18.116541 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:18.112351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:37027 (local address 127.12.12.2:40171)
0621 07:52:18.112566 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:18.112570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:18.112721 (+ 151us) server_negotiation.cc:407] Connection header received
0621 07:52:18.112843 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:18.112846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:18.112894 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:18.113034 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:52:18.113421 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.114019 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.115435 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.115650 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.115739 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:18.116097 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:52:18.116177 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:52:18.116234 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:18.116384 (+ 150us) server_negotiation.cc:299] Negotiation successful
0621 07:52:18.116431 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260621 07:52:18.116603 20421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:18.112237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:40171 (local address 127.12.12.1:37027)
0621 07:52:18.112607 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:18.112623 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:18.112752 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:18.113043 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:18.113046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:18.113054 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:52:18.113193 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:18.113199 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.114131 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.114135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:18.115287 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:18.115300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.115763 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.115767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:18.115868 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:18.115923 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:52:18.116260 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:52:18.116267 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 07:52:18.116334 (+ 67us) client_negotiation.cc:240] Negotiation successful
0621 07:52:18.116427 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":123,"threads_started":1}
I20260621 07:52:18.124361 20293 raft_consensus.cc:1261] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Refusing update from remote peer da12b3e95de94515857e94d262a8b165: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:18.124372 20153 raft_consensus.cc:1261] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Refusing update from remote peer da12b3e95de94515857e94d262a8b165: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:18.124801 20375 consensus_queue.cc:1035] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:18.124981 20397 consensus_queue.cc:1035] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:18.162263 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:18.151601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:53187 (local address 127.12.12.1:42795)
0621 07:52:18.151710 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:18.151715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:18.158858 (+ 7143us) server_negotiation.cc:407] Connection header received
0621 07:52:18.158991 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:18.158994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:18.159052 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:18.159118 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:52:18.159413 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.159915 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.161119 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.161320 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.161401 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:18.161770 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:52:18.161858 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:52:18.161950 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:18.162048 (+ 98us) server_negotiation.cc:299] Negotiation successful
0621 07:52:18.162106 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 07:52:18.162294 20425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:18.151253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:42795 (local address 127.12.12.2:53187)
0621 07:52:18.158713 (+ 7460us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:18.158729 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:18.158879 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:18.159146 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:18.159149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:18.159156 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:52:18.159269 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:18.159275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.160012 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.160015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:18.161000 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:18.161011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.161459 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.161465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:18.161582 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:18.161633 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:52:18.161953 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:52:18.161963 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 07:52:18.162035 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 07:52:18.162125 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7287,"thread_start_us":115,"threads_started":1}
I20260621 07:52:18.188319 19854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:18.188539 19854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:18.190989 19854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c: Sending DeleteTablet for 3 replicas of tablet 146b24241e5946898f367fb6e13ae39c
I20260621 07:52:18.191687 20273 tablet_service.cc:1514] Processing DeleteTablet for tablet 146b24241e5946898f367fb6e13ae39c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:18 UTC) from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:56460
I20260621 07:52:18.191886 20133 tablet_service.cc:1514] Processing DeleteTablet for tablet 146b24241e5946898f367fb6e13ae39c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:18 UTC) from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:54042
I20260621 07:52:18.191951 20390 tablet_replica.cc:330] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: stopping tablet replica
I20260621 07:52:18.192080 20389 tablet_replica.cc:330] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342: stopping tablet replica
I20260621 07:52:18.192080 20390 raft_consensus.cc:2229] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:18.192178 20389 raft_consensus.cc:2229] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:18.192209 20390 raft_consensus.cc:2258] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:18.192296 20389 raft_consensus.cc:2258] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:18.193079 20390 ts_tablet_manager.cc:1905] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:18.193243 20389 ts_tablet_manager.cc:1905] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:18.196287 20390 ts_tablet_manager.cc:1918] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:52:18.196362 20390 log.cc:1196] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/146b24241e5946898f367fb6e13ae39c
I20260621 07:52:18.196621 20390 ts_tablet_manager.cc:1939] T 146b24241e5946898f367fb6e13ae39c P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Deleting consensus metadata
I20260621 07:52:18.198429 19842 catalog_manager.cc:4841] TS 8ae8ba34f4cb4f8db4bf43eff8c00dd8 (127.12.12.3:44809): tablet 146b24241e5946898f367fb6e13ae39c (table test-table [id=dc5428b61f654c7fbcd8b973726b7913]) successfully deleted
I20260621 07:52:18.200403 20389 ts_tablet_manager.cc:1918] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:52:18.200477 20389 log.cc:1196] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/146b24241e5946898f367fb6e13ae39c
I20260621 07:52:18.200726 20389 ts_tablet_manager.cc:1939] T 146b24241e5946898f367fb6e13ae39c P de4777f89fad4de0aac931ffd1538342: Deleting consensus metadata
I20260621 07:52:18.201604 19840 catalog_manager.cc:4841] TS de4777f89fad4de0aac931ffd1538342 (127.12.12.2:40171): tablet 146b24241e5946898f367fb6e13ae39c (table test-table [id=dc5428b61f654c7fbcd8b973726b7913]) successfully deleted
I20260621 07:52:18.201891 19993 tablet_service.cc:1514] Processing DeleteTablet for tablet 146b24241e5946898f367fb6e13ae39c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:18 UTC) from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:40192
I20260621 07:52:18.202111 20391 tablet_replica.cc:330] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165: stopping tablet replica
I20260621 07:52:18.202243 20391 raft_consensus.cc:2229] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:52:18.202647 20391 raft_consensus.cc:2258] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:18.203528 20391 ts_tablet_manager.cc:1905] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:18.206651 20391 ts_tablet_manager.cc:1918] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:52:18.206756 20391 log.cc:1196] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/wal/wals/146b24241e5946898f367fb6e13ae39c
I20260621 07:52:18.207075 20391 ts_tablet_manager.cc:1939] T 146b24241e5946898f367fb6e13ae39c P da12b3e95de94515857e94d262a8b165: Deleting consensus metadata
I20260621 07:52:18.208000 19843 catalog_manager.cc:4841] TS da12b3e95de94515857e94d262a8b165 (127.12.12.1:42795): tablet 146b24241e5946898f367fb6e13ae39c (table test-table [id=dc5428b61f654c7fbcd8b973726b7913]) successfully deleted
Jun 21 07:52:18 dist-test-slave-lbpt krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 07:52:18 dist-test-slave-lbpt krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:18 dist-test-slave-lbpt krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 07:52:18.237353 20405 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:18.228871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.12.12.62:43865)
0621 07:52:18.229029 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:18.229034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:18.229236 (+ 202us) server_negotiation.cc:407] Connection header received
0621 07:52:18.229489 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:18.229493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:18.229554 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:18.229648 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:18.230725 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.231590 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.232962 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.233338 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.233473 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:18.237212 (+ 3739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38254: BlockingRecv error: recv got EOF from 127.0.0.1:38254 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 21 07:52:18 dist-test-slave-lbpt krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM
I20260621 07:52:18.249588 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:18.239583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.12.12.62:43865)
0621 07:52:18.239696 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:18.239700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:18.240152 (+ 452us) server_negotiation.cc:407] Connection header received
0621 07:52:18.240252 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:18.240255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:18.240308 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:18.240394 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:18.241190 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.242004 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.243012 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.243366 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.243475 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:18.246232 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:18.246236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:18.246239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:18.246269 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:18.247896 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:18.248328 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:18.248332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:18.248334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:18.248423 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:18.248695 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:18.248700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:18.248702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:18.248888 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:18.249027 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:18.249279 (+ 252us) server_negotiation.cc:299] Negotiation successful
0621 07:52:18.249416 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:52:18.252491 19854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:52:18.252959 19854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:52:18.258133 20133 tablet_service.cc:1467] Processing CreateTablet for tablet e828862acbc44f77b495b922163c2365 (DEFAULT_TABLE table=test-table [id=001cb923d2bc412e9d88516a554c6d57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:18.258133 19993 tablet_service.cc:1467] Processing CreateTablet for tablet e828862acbc44f77b495b922163c2365 (DEFAULT_TABLE table=test-table [id=001cb923d2bc412e9d88516a554c6d57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:18.258473 20133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e828862acbc44f77b495b922163c2365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:18.258473 19993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e828862acbc44f77b495b922163c2365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:18.258433 20273 tablet_service.cc:1467] Processing CreateTablet for tablet e828862acbc44f77b495b922163c2365 (DEFAULT_TABLE table=test-table [id=001cb923d2bc412e9d88516a554c6d57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:18.258729 20273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e828862acbc44f77b495b922163c2365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:18.260905 20371 tablet_bootstrap.cc:492] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165: Bootstrap starting.
I20260621 07:52:18.261053 20367 tablet_bootstrap.cc:492] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Bootstrap starting.
I20260621 07:52:18.261144 20368 tablet_bootstrap.cc:492] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342: Bootstrap starting.
I20260621 07:52:18.261900 20368 tablet_bootstrap.cc:654] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:18.261931 20367 tablet_bootstrap.cc:654] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:18.261927 20371 tablet_bootstrap.cc:654] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:18.263154 20368 tablet_bootstrap.cc:492] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342: No bootstrap required, opened a new log
I20260621 07:52:18.263234 20368 ts_tablet_manager.cc:1397] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:52:18.263218 20371 tablet_bootstrap.cc:492] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165: No bootstrap required, opened a new log
I20260621 07:52:18.263330 20371 ts_tablet_manager.cc:1397] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260621 07:52:18.263715 20367 tablet_bootstrap.cc:492] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: No bootstrap required, opened a new log
I20260621 07:52:18.263674 20368 raft_consensus.cc:348] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.263780 20368 raft_consensus.cc:374] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:18.263790 20367 ts_tablet_manager.cc:1397] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260621 07:52:18.263814 20368 raft_consensus.cc:729] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4777f89fad4de0aac931ffd1538342, State: Initialized, Role: FOLLOWER
I20260621 07:52:18.263820 20371 raft_consensus.cc:348] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.263933 20371 raft_consensus.cc:374] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:18.263965 20371 raft_consensus.cc:729] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da12b3e95de94515857e94d262a8b165, State: Initialized, Role: FOLLOWER
I20260621 07:52:18.263934 20368 consensus_queue.cc:260] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.264074 20371 consensus_queue.cc:260] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.264182 20368 ts_tablet_manager.cc:1428] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:52:18.264182 20367 raft_consensus.cc:348] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.264279 20367 raft_consensus.cc:374] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:18.264309 20367 raft_consensus.cc:729] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae8ba34f4cb4f8db4bf43eff8c00dd8, State: Initialized, Role: FOLLOWER
I20260621 07:52:18.264343 20371 ts_tablet_manager.cc:1428] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:52:18.264447 20367 consensus_queue.cc:260] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.264730 20367 ts_tablet_manager.cc:1428] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:52:18.323236 20375 raft_consensus.cc:482] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:52:18.323441 20375 raft_consensus.cc:504] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.323949 20375 leader_election.cc:288] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de4777f89fad4de0aac931ffd1538342 (127.12.12.2:40171), 8ae8ba34f4cb4f8db4bf43eff8c00dd8 (127.12.12.3:44809)
I20260621 07:52:18.324424 20153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e828862acbc44f77b495b922163c2365" candidate_uuid: "da12b3e95de94515857e94d262a8b165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de4777f89fad4de0aac931ffd1538342" is_pre_election: true
I20260621 07:52:18.324452 20293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e828862acbc44f77b495b922163c2365" candidate_uuid: "da12b3e95de94515857e94d262a8b165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" is_pre_election: true
I20260621 07:52:18.324605 20153 raft_consensus.cc:2454] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da12b3e95de94515857e94d262a8b165 in term 0.
I20260621 07:52:18.324620 20293 raft_consensus.cc:2454] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da12b3e95de94515857e94d262a8b165 in term 0.
I20260621 07:52:18.324949 19946 leader_election.cc:302] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae8ba34f4cb4f8db4bf43eff8c00dd8, da12b3e95de94515857e94d262a8b165; no voters:
I20260621 07:52:18.325169 20375 raft_consensus.cc:2781] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:52:18.325243 20375 raft_consensus.cc:482] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:52:18.325289 20375 raft_consensus.cc:3037] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:18.326656 20375 raft_consensus.cc:504] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.327104 20375 leader_election.cc:288] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [CANDIDATE]: Term 1 election: Requested vote from peers de4777f89fad4de0aac931ffd1538342 (127.12.12.2:40171), 8ae8ba34f4cb4f8db4bf43eff8c00dd8 (127.12.12.3:44809)
I20260621 07:52:18.327512 20293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e828862acbc44f77b495b922163c2365" candidate_uuid: "da12b3e95de94515857e94d262a8b165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8"
I20260621 07:52:18.327529 20153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e828862acbc44f77b495b922163c2365" candidate_uuid: "da12b3e95de94515857e94d262a8b165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de4777f89fad4de0aac931ffd1538342"
I20260621 07:52:18.327657 20153 raft_consensus.cc:3037] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:18.327661 20293 raft_consensus.cc:3037] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:18.328737 20293 raft_consensus.cc:2454] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da12b3e95de94515857e94d262a8b165 in term 1.
I20260621 07:52:18.328845 20153 raft_consensus.cc:2454] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da12b3e95de94515857e94d262a8b165 in term 1.
I20260621 07:52:18.329063 19946 leader_election.cc:302] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae8ba34f4cb4f8db4bf43eff8c00dd8, da12b3e95de94515857e94d262a8b165; no voters:
I20260621 07:52:18.329350 20375 raft_consensus.cc:2781] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:18.329764 20375 raft_consensus.cc:686] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [term 1 LEADER]: Becoming Leader. State: Replica: da12b3e95de94515857e94d262a8b165, State: Running, Role: LEADER
I20260621 07:52:18.329943 20375 consensus_queue.cc:237] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.331852 19854 catalog_manager.cc:5495] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 reported cstate change: term changed from 0 to 1, leader changed from <none> to da12b3e95de94515857e94d262a8b165 (127.12.12.1). New cstate: current_term: 1 leader_uuid: "da12b3e95de94515857e94d262a8b165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } health_report { overall_health: UNKNOWN } } }
I20260621 07:52:18.353993 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:18.350625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.12.12.1:42795)
0621 07:52:18.350766 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:18.350771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:18.350841 (+ 70us) server_negotiation.cc:407] Connection header received
0621 07:52:18.351013 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:18.351016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:18.351069 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:18.351160 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:52:18.351430 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.351950 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.352860 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:18.353096 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:18.353209 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:18.353507 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:52:18.353593 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:52:18.353676 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:18.353768 (+ 92us) server_negotiation.cc:299] Negotiation successful
0621 07:52:18.353846 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 07:52:18.356243 20293 raft_consensus.cc:1261] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Refusing update from remote peer da12b3e95de94515857e94d262a8b165: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:18.356244 20153 raft_consensus.cc:1261] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Refusing update from remote peer da12b3e95de94515857e94d262a8b165: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:18.356745 20375 consensus_queue.cc:1035] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:18.356956 20426 consensus_queue.cc:1035] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:18.364146 19855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:18.364307 19855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:18.367071 19855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c: Sending DeleteTablet for 3 replicas of tablet e828862acbc44f77b495b922163c2365
I20260621 07:52:18.367660 19993 tablet_service.cc:1514] Processing DeleteTablet for tablet e828862acbc44f77b495b922163c2365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:18 UTC) from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:40192
I20260621 07:52:18.367830 19992 tablet_service.cc:1514] Processing DeleteTablet for tablet e828862acbc44f77b495b922163c2365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:18 UTC) from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:40192
I20260621 07:52:18.367949 20391 tablet_replica.cc:330] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165: stopping tablet replica
I20260621 07:52:18.368062 20391 raft_consensus.cc:2229] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:52:18.368224 20391 raft_consensus.cc:2258] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:18.368400 20133 tablet_service.cc:1514] Processing DeleteTablet for tablet e828862acbc44f77b495b922163c2365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:18 UTC) from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:54042
I20260621 07:52:18.368413 20273 tablet_service.cc:1514] Processing DeleteTablet for tablet e828862acbc44f77b495b922163c2365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:18 UTC) from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:56460
I20260621 07:52:18.368577 20390 tablet_replica.cc:330] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: stopping tablet replica
I20260621 07:52:18.368574 20389 tablet_replica.cc:330] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342: stopping tablet replica
I20260621 07:52:18.368696 20389 raft_consensus.cc:2229] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:18.368698 20390 raft_consensus.cc:2229] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:18.368782 20389 pending_rounds.cc:62] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342: Trying to abort 2 pending ops.
I20260621 07:52:18.368814 20390 raft_consensus.cc:2258] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:18.368860 20389 pending_rounds.cc:68] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7299188073902239744 op_type: WRITE_OP write_request { tablet_id: "e828862acbc44f77b495b922163c2365" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\336\262\336\321\006\"8\n\ntest-admin\022*\n 001cb923d2bc412e9d88516a554c6d57\020\001\030\001 \001(\001" signature: "<redacted>""Q>1\241@?w\221\344\010n\001\363\374\276HA\316\235C\222\374\3074_\314a8\013H\221\340\315\357\006%^gK)\323\372\031\035\240Fm\0207v)\376Y\203+\255[~\343\3160\006mW" signing_key_seq_num: 0 } } request_id { client_id: "8ef34ccf3369415d8dfa1fb6498046c8" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260621 07:52:18.369143 20391 ts_tablet_manager.cc:1905] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:18.369246 20389 pending_rounds.cc:68] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7299188073800892416 op_type: NO_OP noop_request { }
I20260621 07:52:18.369320 20389 raft_consensus.cc:2866] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260621 07:52:18.369390 20389 raft_consensus.cc:2258] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:18.370070 20389 ts_tablet_manager.cc:1905] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:18.370815 19856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:52:18.371217 19856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:52:18.371690 20391 ts_tablet_manager.cc:1918] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:18.371754 20391 log.cc:1196] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/wal/wals/e828862acbc44f77b495b922163c2365
I20260621 07:52:18.372017 20391 ts_tablet_manager.cc:1939] T e828862acbc44f77b495b922163c2365 P da12b3e95de94515857e94d262a8b165: Deleting consensus metadata
I20260621 07:52:18.372532 20389 ts_tablet_manager.cc:1918] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:18.372596 20389 log.cc:1196] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/e828862acbc44f77b495b922163c2365
I20260621 07:52:18.372614 19843 catalog_manager.cc:4841] TS da12b3e95de94515857e94d262a8b165 (127.12.12.1:42795): tablet e828862acbc44f77b495b922163c2365 (table test-table [id=001cb923d2bc412e9d88516a554c6d57]) successfully deleted
I20260621 07:52:18.372860 20389 ts_tablet_manager.cc:1939] T e828862acbc44f77b495b922163c2365 P de4777f89fad4de0aac931ffd1538342: Deleting consensus metadata
I20260621 07:52:18.373281 20390 ts_tablet_manager.cc:1905] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260621 07:52:18.374965 19843 catalog_manager.cc:4808] TS da12b3e95de94515857e94d262a8b165 (127.12.12.1:42795): delete failed for tablet e828862acbc44f77b495b922163c2365 because the tablet was not found. No further retry: Not found: Tablet not found: e828862acbc44f77b495b922163c2365
I20260621 07:52:18.375656 20390 ts_tablet_manager.cc:1918] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:18.375746 20390 log.cc:1196] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/e828862acbc44f77b495b922163c2365
I20260621 07:52:18.376005 20390 ts_tablet_manager.cc:1939] T e828862acbc44f77b495b922163c2365 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Deleting consensus metadata
I20260621 07:52:18.376369 19992 tablet_service.cc:1467] Processing CreateTablet for tablet f55ca29d2ea540e4b549835132b7cca7 (DEFAULT_TABLE table=test-table [id=8930c5d8ac2f42aebd7dfe6a3139046a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:18.376374 20273 tablet_service.cc:1467] Processing CreateTablet for tablet f55ca29d2ea540e4b549835132b7cca7 (DEFAULT_TABLE table=test-table [id=8930c5d8ac2f42aebd7dfe6a3139046a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:18.376591 19992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55ca29d2ea540e4b549835132b7cca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:18.376637 19840 catalog_manager.cc:4841] TS de4777f89fad4de0aac931ffd1538342 (127.12.12.2:40171): tablet e828862acbc44f77b495b922163c2365 (table test-table [id=001cb923d2bc412e9d88516a554c6d57]) successfully deleted
I20260621 07:52:18.376909 20133 tablet_service.cc:1467] Processing CreateTablet for tablet f55ca29d2ea540e4b549835132b7cca7 (DEFAULT_TABLE table=test-table [id=8930c5d8ac2f42aebd7dfe6a3139046a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:18.377112 20133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55ca29d2ea540e4b549835132b7cca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:18.378553 20371 tablet_bootstrap.cc:492] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165: Bootstrap starting.
I20260621 07:52:18.379040 20368 tablet_bootstrap.cc:492] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342: Bootstrap starting.
I20260621 07:52:18.379400 20371 tablet_bootstrap.cc:654] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:18.379892 20368 tablet_bootstrap.cc:654] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:18.380508 20273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55ca29d2ea540e4b549835132b7cca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:18.380985 20368 tablet_bootstrap.cc:492] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342: No bootstrap required, opened a new log
I20260621 07:52:18.381058 20368 ts_tablet_manager.cc:1397] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:52:18.381381 20368 raft_consensus.cc:348] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.381490 20368 raft_consensus.cc:374] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:18.381526 20368 raft_consensus.cc:729] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4777f89fad4de0aac931ffd1538342, State: Initialized, Role: FOLLOWER
I20260621 07:52:18.381651 20368 consensus_queue.cc:260] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.381910 20368 ts_tablet_manager.cc:1428] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260621 07:52:18.382440 20371 tablet_bootstrap.cc:492] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165: No bootstrap required, opened a new log
I20260621 07:52:18.382524 20371 ts_tablet_manager.cc:1397] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260621 07:52:18.382951 20371 raft_consensus.cc:348] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.383070 20371 raft_consensus.cc:374] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:18.383111 20371 raft_consensus.cc:729] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da12b3e95de94515857e94d262a8b165, State: Initialized, Role: FOLLOWER
I20260621 07:52:18.383229 20371 consensus_queue.cc:260] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.383342 19842 catalog_manager.cc:4841] TS 8ae8ba34f4cb4f8db4bf43eff8c00dd8 (127.12.12.3:44809): tablet e828862acbc44f77b495b922163c2365 (table test-table [id=001cb923d2bc412e9d88516a554c6d57]) successfully deleted
I20260621 07:52:18.383510 20371 ts_tablet_manager.cc:1428] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:52:18.383746 20367 tablet_bootstrap.cc:492] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Bootstrap starting.
I20260621 07:52:18.384442 20367 tablet_bootstrap.cc:654] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:18.386296 20367 tablet_bootstrap.cc:492] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: No bootstrap required, opened a new log
I20260621 07:52:18.386394 20367 ts_tablet_manager.cc:1397] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260621 07:52:18.386819 20367 raft_consensus.cc:348] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.386950 20367 raft_consensus.cc:374] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:18.386993 20367 raft_consensus.cc:729] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae8ba34f4cb4f8db4bf43eff8c00dd8, State: Initialized, Role: FOLLOWER
I20260621 07:52:18.387130 20367 consensus_queue.cc:260] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.387391 20367 ts_tablet_manager.cc:1428] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260621 07:52:18.490458 20375 raft_consensus.cc:482] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:52:18.490603 20375 raft_consensus.cc:504] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.490990 20375 leader_election.cc:288] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de4777f89fad4de0aac931ffd1538342 (127.12.12.2:40171), 8ae8ba34f4cb4f8db4bf43eff8c00dd8 (127.12.12.3:44809)
I20260621 07:52:18.491601 20293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55ca29d2ea540e4b549835132b7cca7" candidate_uuid: "da12b3e95de94515857e94d262a8b165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" is_pre_election: true
I20260621 07:52:18.491601 20153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55ca29d2ea540e4b549835132b7cca7" candidate_uuid: "da12b3e95de94515857e94d262a8b165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de4777f89fad4de0aac931ffd1538342" is_pre_election: true
I20260621 07:52:18.491771 20153 raft_consensus.cc:2454] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da12b3e95de94515857e94d262a8b165 in term 0.
I20260621 07:52:18.491771 20293 raft_consensus.cc:2454] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da12b3e95de94515857e94d262a8b165 in term 0.
I20260621 07:52:18.492110 19946 leader_election.cc:302] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae8ba34f4cb4f8db4bf43eff8c00dd8, da12b3e95de94515857e94d262a8b165; no voters:
I20260621 07:52:18.492314 20375 raft_consensus.cc:2781] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:52:18.492385 20375 raft_consensus.cc:482] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:52:18.492413 20375 raft_consensus.cc:3037] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:18.493453 20375 raft_consensus.cc:504] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.493834 20375 leader_election.cc:288] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [CANDIDATE]: Term 1 election: Requested vote from peers de4777f89fad4de0aac931ffd1538342 (127.12.12.2:40171), 8ae8ba34f4cb4f8db4bf43eff8c00dd8 (127.12.12.3:44809)
I20260621 07:52:18.494295 20153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55ca29d2ea540e4b549835132b7cca7" candidate_uuid: "da12b3e95de94515857e94d262a8b165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de4777f89fad4de0aac931ffd1538342"
I20260621 07:52:18.494385 20293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55ca29d2ea540e4b549835132b7cca7" candidate_uuid: "da12b3e95de94515857e94d262a8b165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8"
I20260621 07:52:18.494491 20153 raft_consensus.cc:3037] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:18.494549 20293 raft_consensus.cc:3037] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:18.495464 20153 raft_consensus.cc:2454] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da12b3e95de94515857e94d262a8b165 in term 1.
I20260621 07:52:18.495486 20293 raft_consensus.cc:2454] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da12b3e95de94515857e94d262a8b165 in term 1.
I20260621 07:52:18.495814 19944 leader_election.cc:302] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da12b3e95de94515857e94d262a8b165, de4777f89fad4de0aac931ffd1538342; no voters:
I20260621 07:52:18.496065 20375 raft_consensus.cc:2781] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:18.496158 20375 raft_consensus.cc:686] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [term 1 LEADER]: Becoming Leader. State: Replica: da12b3e95de94515857e94d262a8b165, State: Running, Role: LEADER
I20260621 07:52:18.496305 20375 consensus_queue.cc:237] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } }
I20260621 07:52:18.498003 19856 catalog_manager.cc:5495] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 reported cstate change: term changed from 0 to 1, leader changed from <none> to da12b3e95de94515857e94d262a8b165 (127.12.12.1). New cstate: current_term: 1 leader_uuid: "da12b3e95de94515857e94d262a8b165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da12b3e95de94515857e94d262a8b165" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 } health_report { overall_health: UNKNOWN } } }
I20260621 07:52:18.521162 20293 raft_consensus.cc:1261] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Refusing update from remote peer da12b3e95de94515857e94d262a8b165: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:18.521282 20153 raft_consensus.cc:1261] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Refusing update from remote peer da12b3e95de94515857e94d262a8b165: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:18.521653 20397 consensus_queue.cc:1035] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae8ba34f4cb4f8db4bf43eff8c00dd8" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 44809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:18.521813 20375 consensus_queue.cc:1035] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de4777f89fad4de0aac931ffd1538342" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:18.554203 19855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:18.554355 19855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:18.556697 19855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d942cdbfceb14a9ea6522ec4d98a191c: Sending DeleteTablet for 3 replicas of tablet f55ca29d2ea540e4b549835132b7cca7
I20260621 07:52:18.557281 19992 tablet_service.cc:1514] Processing DeleteTablet for tablet f55ca29d2ea540e4b549835132b7cca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:18 UTC) from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:40192
I20260621 07:52:18.557399 20133 tablet_service.cc:1514] Processing DeleteTablet for tablet f55ca29d2ea540e4b549835132b7cca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:18 UTC) from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:54042
I20260621 07:52:18.557482 20391 tablet_replica.cc:330] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165: stopping tablet replica
I20260621 07:52:18.557520 20273 tablet_service.cc:1514] Processing DeleteTablet for tablet f55ca29d2ea540e4b549835132b7cca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:18 UTC) from {username='oryx', principal='oryx/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:56460
I20260621 07:52:18.557582 20389 tablet_replica.cc:330] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342: stopping tablet replica
I20260621 07:52:18.557601 20391 raft_consensus.cc:2229] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:52:18.557682 20390 tablet_replica.cc:330] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: stopping tablet replica
I20260621 07:52:18.557690 20389 raft_consensus.cc:2229] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:18.557796 20391 raft_consensus.cc:2258] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:18.557776 20390 raft_consensus.cc:2229] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:18.557814 20389 raft_consensus.cc:2258] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:18.557917 20390 raft_consensus.cc:2258] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:18.558660 20391 ts_tablet_manager.cc:1905] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:18.558712 20390 ts_tablet_manager.cc:1905] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:18.558867 20389 ts_tablet_manager.cc:1905] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:18.560829 20391 ts_tablet_manager.cc:1918] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:52:18.560927 20391 log.cc:1196] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-0/wal/wals/f55ca29d2ea540e4b549835132b7cca7
I20260621 07:52:18.561193 20391 ts_tablet_manager.cc:1939] T f55ca29d2ea540e4b549835132b7cca7 P da12b3e95de94515857e94d262a8b165: Deleting consensus metadata
I20260621 07:52:18.561226 20390 ts_tablet_manager.cc:1918] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:52:18.561312 20390 log.cc:1196] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/f55ca29d2ea540e4b549835132b7cca7
I20260621 07:52:18.561322 20389 ts_tablet_manager.cc:1918] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:52:18.561401 20389 log.cc:1196] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/f55ca29d2ea540e4b549835132b7cca7
I20260621 07:52:18.561589 20390 ts_tablet_manager.cc:1939] T f55ca29d2ea540e4b549835132b7cca7 P 8ae8ba34f4cb4f8db4bf43eff8c00dd8: Deleting consensus metadata
I20260621 07:52:18.561650 20389 ts_tablet_manager.cc:1939] T f55ca29d2ea540e4b549835132b7cca7 P de4777f89fad4de0aac931ffd1538342: Deleting consensus metadata
I20260621 07:52:18.562013 19843 catalog_manager.cc:4841] TS da12b3e95de94515857e94d262a8b165 (127.12.12.1:42795): tablet f55ca29d2ea540e4b549835132b7cca7 (table test-table [id=8930c5d8ac2f42aebd7dfe6a3139046a]) successfully deleted
I20260621 07:52:18.562219 19842 catalog_manager.cc:4841] TS 8ae8ba34f4cb4f8db4bf43eff8c00dd8 (127.12.12.3:44809): tablet f55ca29d2ea540e4b549835132b7cca7 (table test-table [id=8930c5d8ac2f42aebd7dfe6a3139046a]) successfully deleted
I20260621 07:52:18.562418 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 19928
I20260621 07:52:18.562547 19840 catalog_manager.cc:4841] TS de4777f89fad4de0aac931ffd1538342 (127.12.12.2:40171): tablet f55ca29d2ea540e4b549835132b7cca7 (table test-table [id=8930c5d8ac2f42aebd7dfe6a3139046a]) successfully deleted
I20260621 07:52:18.570207 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 20068
I20260621 07:52:18.577296 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 20208
I20260621 07:52:18.584268 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 19824
2026-06-21T07:52:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3358 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:52:18 dist-test-slave-lbpt krb5kdc[20456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:52:18 dist-test-slave-lbpt krb5kdc[20456](info): set up 2 sockets
Jun 21 07:52:18 dist-test-slave-lbpt krb5kdc[20456](info): commencing operation
krb5kdc: starting...
W20260621 07:52:19.647312 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:52:19 dist-test-slave-lbpt krb5kdc[20456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:52:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:52:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:52:19.771772 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:39953
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:39953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:19.885533 20469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:19.885834 20469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:19.885877 20469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:19.889899 20469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:19.889966 20469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:19.889986 20469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:19.890003 20469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:19.890018 20469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:52:19.895373 20469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:39953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:39953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:19.896513 20469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:19.897531 20469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:19.906112 20474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:19.906150 20475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:19.906172 20477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:19.906345 20469 server_base.cc:1034] running on GCE node
I20260621 07:52:19.907094 20469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:19.908349 20469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:19.909555 20469 hybrid_clock.cc:648] HybridClock initialized: now 1782028339909533 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-21T07:52:19Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1333 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:52:19 dist-test-slave-lbpt krb5kdc[20482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:52:19 dist-test-slave-lbpt krb5kdc[20482](info): set up 2 sockets
Jun 21 07:52:19 dist-test-slave-lbpt krb5kdc[20482](info): commencing operation
krb5kdc: starting...
W20260621 07:52:20.959357 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:52:21 dist-test-slave-lbpt krb5kdc[20482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:52:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:52:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:52:21.084806 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:38533
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:38533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:21.196101 20495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:21.196462 20495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:21.196547 20495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:21.200657 20495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:21.200773 20495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:21.200814 20495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:21.200847 20495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:21.200875 20495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:52:21.206610 20495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:38533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:38533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:21.207814 20495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:21.208902 20495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:21.217943 20501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:21.217947 20500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:21.217947 20503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:21.218400 20495 server_base.cc:1034] running on GCE node
I20260621 07:52:21.219074 20495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:21.220196 20495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:21.221406 20495 hybrid_clock.cc:648] HybridClock initialized: now 1782028341221390 us; error 38 us; skew 500 ppm
Jun 21 07:52:21 dist-test-slave-lbpt krb5kdc[20482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028341, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:21.228202 20495 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:52:21.229215 20495 webserver.cc:460] Webserver started at http://127.12.12.62:35337/ using document root <none> and password file <none>
I20260621 07:52:21.229770 20495 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:21.229835 20495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:21.230018 20495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:21.232254 20495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "ae7bfe939f7348518939eddc284d3bd9"
format_stamp: "Formatted at 2026-06-21 07:52:21 on dist-test-slave-lbpt"
server_key: "11a64bdfdf8e563bab1374af6a40c8c0"
server_key_iv: "1a8914b1809a79b041a00a6b66b59a3e"
server_key_version: "encryptionkey@0"
I20260621 07:52:21.232798 20495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "ae7bfe939f7348518939eddc284d3bd9"
format_stamp: "Formatted at 2026-06-21 07:52:21 on dist-test-slave-lbpt"
server_key: "11a64bdfdf8e563bab1374af6a40c8c0"
server_key_iv: "1a8914b1809a79b041a00a6b66b59a3e"
server_key_version: "encryptionkey@0"
I20260621 07:52:21.236742 20495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260621 07:52:21.239277 20510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:21.240598 20495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260621 07:52:21.240761 20495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "ae7bfe939f7348518939eddc284d3bd9"
format_stamp: "Formatted at 2026-06-21 07:52:21 on dist-test-slave-lbpt"
server_key: "11a64bdfdf8e563bab1374af6a40c8c0"
server_key_iv: "1a8914b1809a79b041a00a6b66b59a3e"
server_key_version: "encryptionkey@0"
I20260621 07:52:21.240885 20495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:21.276924 20495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:21.280233 20495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:21.280397 20495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:21.287742 20495 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:38533
I20260621 07:52:21.287810 20562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:38533 every 8 connection(s)
I20260621 07:52:21.288744 20495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:52:21.291713 20563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:21.291810 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 20495
I20260621 07:52:21.291980 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:52:21.292309 12336 external_mini_cluster.cc:1428] Setting key 3b8c61f5f5a47c1181395e85406ae2ea
I20260621 07:52:21.298208 20563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9: Bootstrap starting.
I20260621 07:52:21.300664 20563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9: Neither blocks nor log segments found. Creating new log.
Jun 21 07:52:21 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:21.301472 20563 log.cc:824] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:21.303925 20563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9: No bootstrap required, opened a new log
I20260621 07:52:21.307165 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:21.293701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.12.12.62:38533)
0621 07:52:21.294149 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:21.294164 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:21.294199 (+ 35us) server_negotiation.cc:407] Connection header received
0621 07:52:21.295132 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:21.295151 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:21.295488 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:21.295846 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:21.296647 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:21.297615 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:21.298598 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:21.299077 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:21.299255 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:21.301874 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:21.301879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:21.301894 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:21.301921 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:21.303874 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:21.304377 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:21.304383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:21.304389 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:21.304528 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:21.304785 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:21.304789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:21.304790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:21.305291 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:21.305486 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:21.305744 (+ 258us) server_negotiation.cc:299] Negotiation successful
0621 07:52:21.306074 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":137,"threads_started":1}
I20260621 07:52:21.308569 20563 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7bfe939f7348518939eddc284d3bd9" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38533 } }
I20260621 07:52:21.308818 20563 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:21.308883 20563 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7bfe939f7348518939eddc284d3bd9, State: Initialized, Role: FOLLOWER
I20260621 07:52:21.309325 20563 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7bfe939f7348518939eddc284d3bd9" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38533 } }
I20260621 07:52:21.309496 20563 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:52:21.309592 20563 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:52:21.309712 20563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:21.310743 20563 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7bfe939f7348518939eddc284d3bd9" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38533 } }
I20260621 07:52:21.311082 20563 leader_election.cc:302] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae7bfe939f7348518939eddc284d3bd9; no voters:
I20260621 07:52:21.311398 20563 leader_election.cc:288] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:52:21.311703 20568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:21.311905 20568 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 1 LEADER]: Becoming Leader. State: Replica: ae7bfe939f7348518939eddc284d3bd9, State: Running, Role: LEADER
I20260621 07:52:21.312250 20568 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7bfe939f7348518939eddc284d3bd9" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38533 } }
I20260621 07:52:21.312644 20563 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:52:21.314097 20569 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae7bfe939f7348518939eddc284d3bd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7bfe939f7348518939eddc284d3bd9" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38533 } } }
I20260621 07:52:21.314249 20569 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:21.314103 20570 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae7bfe939f7348518939eddc284d3bd9. Latest consensus state: current_term: 1 leader_uuid: "ae7bfe939f7348518939eddc284d3bd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7bfe939f7348518939eddc284d3bd9" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38533 } } }
I20260621 07:52:21.315260 20570 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [sys.catalog]: This master's current role is: LEADER
W20260621 07:52:21.317782 20583 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:52:21.317862 20583 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:52:21.317998 20580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:52:21.318845 20580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:52:21.324592 20580 catalog_manager.cc:1347] Generated new cluster ID: a4e0d8ccbc0b44daa580ababcabc01a6
I20260621 07:52:21.324693 20580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:52:21.332551 20580 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:52:21.333527 20580 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:52:21.340353 20580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9: Generated new TSK 0
I20260621 07:52:21.341068 20580 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:52:21.371893 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38533
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:21.490428 20589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:21.490723 20589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:21.490769 20589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:21.494925 20589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:21.494987 20589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:21.495105 20589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:52:21.500516 20589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:21.501642 20589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:21.502758 20589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:52:21.513048 20589 server_base.cc:1034] running on GCE node
W20260621 07:52:21.512981 20595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:21.512984 20594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:21.512981 20597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:21.513824 20589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:21.514588 20589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:21.515789 20589 hybrid_clock.cc:648] HybridClock initialized: now 1782028341515754 us; error 48 us; skew 500 ppm
Jun 21 07:52:21 dist-test-slave-lbpt krb5kdc[20482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028341, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:21.518873 20589 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:52:21.519804 20589 webserver.cc:460] Webserver started at http://127.12.12.1:40811/ using document root <none> and password file <none>
I20260621 07:52:21.520411 20589 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:21.520525 20589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:21.520767 20589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:21.522974 20589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "30ca3272b3f843c886227d5dfcc4bc8b"
format_stamp: "Formatted at 2026-06-21 07:52:21 on dist-test-slave-lbpt"
server_key: "deeb6882b1b32ed5770afc7e0a8d7ba8"
server_key_iv: "8d9a56a06956dbf7e0cba2e56b9c9bea"
server_key_version: "encryptionkey@0"
I20260621 07:52:21.523543 20589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "30ca3272b3f843c886227d5dfcc4bc8b"
format_stamp: "Formatted at 2026-06-21 07:52:21 on dist-test-slave-lbpt"
server_key: "deeb6882b1b32ed5770afc7e0a8d7ba8"
server_key_iv: "8d9a56a06956dbf7e0cba2e56b9c9bea"
server_key_version: "encryptionkey@0"
I20260621 07:52:21.527388 20589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:52:21.529884 20604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:21.531268 20589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:52:21.531415 20589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "30ca3272b3f843c886227d5dfcc4bc8b"
format_stamp: "Formatted at 2026-06-21 07:52:21 on dist-test-slave-lbpt"
server_key: "deeb6882b1b32ed5770afc7e0a8d7ba8"
server_key_iv: "8d9a56a06956dbf7e0cba2e56b9c9bea"
server_key_version: "encryptionkey@0"
I20260621 07:52:21.531541 20589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:21.550321 20589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:21.554044 20589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:21.554272 20589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:21.554944 20589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:21.555927 20589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:21.556003 20589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:21.556073 20589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:21.556124 20589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:21.565645 20589 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:40829
I20260621 07:52:21.565663 20717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:40829 every 8 connection(s)
I20260621 07:52:21.566679 20589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:52:21.568936 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 20589
I20260621 07:52:21.569090 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:52:21.569481 12336 external_mini_cluster.cc:1428] Setting key f4c142a89b9904ff5d20d65420a75182
Jun 21 07:52:21 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028341, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
I20260621 07:52:21.583220 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:21.568575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:59835 (local address 127.12.12.62:38533)
0621 07:52:21.568783 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:21.568788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:21.570053 (+ 1265us) server_negotiation.cc:407] Connection header received
0621 07:52:21.571123 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:21.571125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:21.571185 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:21.571268 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:21.572994 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:21.573886 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:21.575706 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:21.576062 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:21.576162 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:21.579410 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:21.579414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:21.579416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:21.579440 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:21.580978 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:21.581536 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:21.581540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:21.581542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:21.581625 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:21.581992 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:21.581995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:21.581997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:21.582190 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:21.582301 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:21.582913 (+ 612us) server_negotiation.cc:299] Negotiation successful
0621 07:52:21.583053 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:52:21.584165 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:21.569006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:59835)
0621 07:52:21.569850 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:21.569912 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:21.570875 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:21.571454 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:21.571466 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:21.572031 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:21.572764 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:21.572789 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:21.574038 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:21.574040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:21.575548 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:21.575563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:21.576257 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:21.576263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:21.576534 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:21.577230 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:21.577259 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:21.579210 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:21.581124 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:21.581131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:21.581148 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:21.581418 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:21.581723 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:21.581727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:21.581729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:21.581885 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:21.582301 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:21.582309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:21.582658 (+ 349us) client_negotiation.cc:769] Sending connection context
0621 07:52:21.582859 (+ 201us) client_negotiation.cc:240] Negotiation successful
0621 07:52:21.583216 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":138,"threads_started":1}
I20260621 07:52:21.585625 20718 heartbeater.cc:347] Connected to a master server at 127.12.12.62:38533
I20260621 07:52:21.585886 20718 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:21.586520 20718 heartbeater.cc:511] Master 127.12.12.62:38533 requested a full tablet report, sending...
I20260621 07:52:21.588382 20527 ts_manager.cc:194] Registered new tserver with Master: 30ca3272b3f843c886227d5dfcc4bc8b (127.12.12.1:40829)
I20260621 07:52:21.590117 20527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:59835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:52:21.601307 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38533
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:21.715586 20723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:21.715868 20723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:21.715945 20723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:21.720203 20723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:21.720276 20723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:21.720337 20723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:52:21.725833 20723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:21.727183 20723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:21.728269 20723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:21.738576 20731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:21.738651 20729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:21.738883 20723 server_base.cc:1034] running on GCE node
W20260621 07:52:21.738950 20728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:21.739410 20723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:21.740093 20723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:21.741255 20723 hybrid_clock.cc:648] HybridClock initialized: now 1782028341741239 us; error 28 us; skew 500 ppm
Jun 21 07:52:21 dist-test-slave-lbpt krb5kdc[20482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028341, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:21.744406 20723 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:52:21.745446 20723 webserver.cc:460] Webserver started at http://127.12.12.2:36699/ using document root <none> and password file <none>
I20260621 07:52:21.746018 20723 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:21.746078 20723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:21.746243 20723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:21.748406 20723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "06e87cb9f19f4a718949e14729646f59"
format_stamp: "Formatted at 2026-06-21 07:52:21 on dist-test-slave-lbpt"
server_key: "00b92dcbdba1643470b9a99f7d923086"
server_key_iv: "e6766b0bc6a5fd1f842e1b22ec57dde6"
server_key_version: "encryptionkey@0"
I20260621 07:52:21.748924 20723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "06e87cb9f19f4a718949e14729646f59"
format_stamp: "Formatted at 2026-06-21 07:52:21 on dist-test-slave-lbpt"
server_key: "00b92dcbdba1643470b9a99f7d923086"
server_key_iv: "e6766b0bc6a5fd1f842e1b22ec57dde6"
server_key_version: "encryptionkey@0"
I20260621 07:52:21.752761 20723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:52:21.755187 20738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:21.756515 20723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:52:21.756662 20723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "06e87cb9f19f4a718949e14729646f59"
format_stamp: "Formatted at 2026-06-21 07:52:21 on dist-test-slave-lbpt"
server_key: "00b92dcbdba1643470b9a99f7d923086"
server_key_iv: "e6766b0bc6a5fd1f842e1b22ec57dde6"
server_key_version: "encryptionkey@0"
I20260621 07:52:21.756783 20723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:21.785867 20723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:21.789443 20723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:21.789666 20723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:21.790302 20723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:21.791211 20723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:21.791288 20723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:21.791360 20723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:21.791411 20723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:21.800865 20723 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:39205
I20260621 07:52:21.800886 20851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:39205 every 8 connection(s)
I20260621 07:52:21.801877 20723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:52:21.808269 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 20723
I20260621 07:52:21.808468 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:52:21.808751 12336 external_mini_cluster.cc:1428] Setting key 2a9307e1f18b4e1e5a9383b557b81aac
Jun 21 07:52:21 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028341, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:21.817123 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:21.803882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:38695 (local address 127.12.12.62:38533)
0621 07:52:21.804023 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:21.804027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:21.805155 (+ 1128us) server_negotiation.cc:407] Connection header received
0621 07:52:21.806152 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:21.806155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:21.806213 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:21.806344 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:21.807573 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:21.808114 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:21.809570 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:21.809876 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:21.809996 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:21.812857 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:21.812865 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:21.812868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:21.812894 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:21.814826 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:21.815467 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:21.815471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:21.815473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:21.815565 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:21.815958 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:21.815961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:21.815963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:21.816171 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:21.816282 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:21.816820 (+ 538us) server_negotiation.cc:299] Negotiation successful
0621 07:52:21.816952 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:52:21.818115 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:21.804290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:38695)
0621 07:52:21.804988 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:21.805032 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:21.805912 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:21.806539 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:21.806546 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:21.806929 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:21.807411 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:21.807426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:21.808260 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:21.808264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:21.809395 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:21.809406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:21.810074 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:21.810081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:21.810272 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:21.810986 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:21.811002 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:21.812669 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:21.815003 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:21.815010 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:21.815028 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:21.815310 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:21.815674 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:21.815678 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:21.815680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:21.815837 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:21.816293 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:21.816301 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:21.816573 (+ 272us) client_negotiation.cc:769] Sending connection context
0621 07:52:21.816779 (+ 206us) client_negotiation.cc:240] Negotiation successful
0621 07:52:21.817149 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
I20260621 07:52:21.819401 20852 heartbeater.cc:347] Connected to a master server at 127.12.12.62:38533
I20260621 07:52:21.819761 20852 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:21.820289 20852 heartbeater.cc:511] Master 127.12.12.62:38533 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
I20260621 07:52:21.821724 20527 ts_manager.cc:194] Registered new tserver with Master: 06e87cb9f19f4a718949e14729646f59 (127.12.12.2:39205)
I20260621 07:52:21.822623 20527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:38695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:52:21.838008 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38533
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:21.953506 20857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:21.953791 20857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:21.953923 20857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:21.958195 20857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:21.958278 20857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:21.958341 20857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:52:21.964047 20857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:21.965178 20857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:21.966245 20857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:21.976161 20863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:21.976207 20862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:21.976202 20865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:21.976398 20857 server_base.cc:1034] running on GCE node
I20260621 07:52:21.976897 20857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:21.977604 20857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:21.978842 20857 hybrid_clock.cc:648] HybridClock initialized: now 1782028341978825 us; error 30 us; skew 500 ppm
Jun 21 07:52:21 dist-test-slave-lbpt krb5kdc[20482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028341, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:21.981696 20857 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:52:21.982671 20857 webserver.cc:460] Webserver started at http://127.12.12.3:44455/ using document root <none> and password file <none>
I20260621 07:52:21.983240 20857 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:21.983337 20857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:21.983573 20857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:21.985808 20857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "9a7523a3fbd647f2bab7f770ad71221d"
format_stamp: "Formatted at 2026-06-21 07:52:21 on dist-test-slave-lbpt"
server_key: "546e6442d80a6aaa0272799250b31b0d"
server_key_iv: "36cde8f0ee63e7a098298c91f74027df"
server_key_version: "encryptionkey@0"
I20260621 07:52:21.986379 20857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "9a7523a3fbd647f2bab7f770ad71221d"
format_stamp: "Formatted at 2026-06-21 07:52:21 on dist-test-slave-lbpt"
server_key: "546e6442d80a6aaa0272799250b31b0d"
server_key_iv: "36cde8f0ee63e7a098298c91f74027df"
server_key_version: "encryptionkey@0"
I20260621 07:52:21.990182 20857 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:52:21.993041 20872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:21.994771 20857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:21.994920 20857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "9a7523a3fbd647f2bab7f770ad71221d"
format_stamp: "Formatted at 2026-06-21 07:52:21 on dist-test-slave-lbpt"
server_key: "546e6442d80a6aaa0272799250b31b0d"
server_key_iv: "36cde8f0ee63e7a098298c91f74027df"
server_key_version: "encryptionkey@0"
I20260621 07:52:21.995054 20857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:22.018798 20857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:22.023156 20857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:22.023324 20857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:22.023910 20857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:22.024725 20857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:22.024772 20857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:22.024816 20857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:22.024840 20857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:22.034152 20857 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:33441
I20260621 07:52:22.034173 20985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:33441 every 8 connection(s)
I20260621 07:52:22.035212 20857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:52:22.035579 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 20857
I20260621 07:52:22.035732 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:52:22.036105 12336 external_mini_cluster.cc:1428] Setting key 7e444e68f2204080285853b87a993127
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028341, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:22.050660 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:22.037376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:59841 (local address 127.12.12.62:38533)
0621 07:52:22.037515 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:22.037520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:22.038781 (+ 1261us) server_negotiation.cc:407] Connection header received
0621 07:52:22.039900 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:22.039903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:22.039961 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:22.040040 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:22.041286 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.041906 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.043189 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.043468 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.043563 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.046448 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:22.046452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:22.046454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:22.046486 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:22.048021 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:22.048610 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:22.048614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:22.048616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:22.048741 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:22.049281 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:22.049285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:22.049287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:22.049529 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:22.049613 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:22.050374 (+ 761us) server_negotiation.cc:299] Negotiation successful
0621 07:52:22.050540 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 07:52:22.051503 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:22.037859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:59841)
0621 07:52:22.038629 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:22.038683 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:22.039502 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:22.040224 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:22.040231 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:22.040605 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:22.041106 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.041120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.042029 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.042032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.043055 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.043064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.043611 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.043614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.043756 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.044553 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:22.044570 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:22.046231 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:22.048167 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:22.048173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:22.048189 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:22.048494 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:22.048968 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:22.048972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:22.048973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:22.049167 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:22.049646 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:22.049656 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:22.050001 (+ 345us) client_negotiation.cc:769] Sending connection context
0621 07:52:22.050221 (+ 220us) client_negotiation.cc:240] Negotiation successful
0621 07:52:22.050644 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":165,"threads_started":1}
I20260621 07:52:22.052660 20986 heartbeater.cc:347] Connected to a master server at 127.12.12.62:38533
I20260621 07:52:22.052964 20986 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:22.053490 20986 heartbeater.cc:511] Master 127.12.12.62:38533 requested a full tablet report, sending...
I20260621 07:52:22.054592 20527 ts_manager.cc:194] Registered new tserver with Master: 9a7523a3fbd647f2bab7f770ad71221d (127.12.12.3:33441)
I20260621 07:52:22.055127 20527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:59841
I20260621 07:52:22.062388 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.1.keytab.
Entry for principal oryx/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.1.keytab.
WARNING: no policy specified for oryx/127.12.12.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.2.keytab.
Entry for principal oryx/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.2.keytab.
WARNING: no policy specified for oryx/127.12.12.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.3.keytab.
Entry for principal oryx/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.3.keytab.
I20260621 07:52:22.140719 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 20589
I20260621 07:52:22.146699 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 20723
I20260621 07:52:22.152168 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 20857
I20260621 07:52:22.157529 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 20495
I20260621 07:52:22.164079 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:38533
--webserver_interface=127.12.12.62
--webserver_port=35337
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:38533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:22.277616 20995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:22.277951 20995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:22.278033 20995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:22.282110 20995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:22.282218 20995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:22.282258 20995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:22.282292 20995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:22.282330 20995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:52:22.287758 20995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:38533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:38533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=35337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:22.288878 20995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:22.289899 20995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:52:22.298477 20995 server_base.cc:1034] running on GCE node
W20260621 07:52:22.298427 21001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:22.298441 21000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:22.298681 21003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:22.299273 20995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:22.300643 20995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:22.301882 20995 hybrid_clock.cc:648] HybridClock initialized: now 1782028342301851 us; error 49 us; skew 500 ppm
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028342, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:22.304771 20995 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:52:22.305589 20995 webserver.cc:460] Webserver started at http://127.12.12.62:35337/ using document root <none> and password file <none>
I20260621 07:52:22.306167 20995 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:22.306290 20995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:22.315052 20995 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.001s
I20260621 07:52:22.317394 21010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:22.318776 20995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:52:22.318909 20995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "ae7bfe939f7348518939eddc284d3bd9"
format_stamp: "Formatted at 2026-06-21 07:52:21 on dist-test-slave-lbpt"
server_key: "11a64bdfdf8e563bab1374af6a40c8c0"
server_key_iv: "1a8914b1809a79b041a00a6b66b59a3e"
server_key_version: "encryptionkey@0"
I20260621 07:52:22.319463 20995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:22.335408 20995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:22.338604 20995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:22.338809 20995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:22.346952 20995 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:38533
I20260621 07:52:22.347044 21062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:38533 every 8 connection(s)
I20260621 07:52:22.348203 20995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:52:22.350980 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 20995
I20260621 07:52:22.351425 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:40829
--local_ip_for_outbound_sockets=127.12.12.1
--tserver_master_addrs=127.12.12.62:38533
--webserver_port=40811
--webserver_interface=127.12.12.1
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.12.1
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:52:22.352957 21063 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:52:22.355341 21063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9: Bootstrap starting.
I20260621 07:52:22.378959 21063 log.cc:824] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:22.387634 21063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:52:22.388057 21063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9: Bootstrap complete.
I20260621 07:52:22.393730 21063 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7bfe939f7348518939eddc284d3bd9" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38533 } }
I20260621 07:52:22.394270 21063 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7bfe939f7348518939eddc284d3bd9, State: Initialized, Role: FOLLOWER
I20260621 07:52:22.394884 21063 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7bfe939f7348518939eddc284d3bd9" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38533 } }
I20260621 07:52:22.395042 21063 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:52:22.395133 21063 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:52:22.395233 21063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:52:22.397894 21063 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7bfe939f7348518939eddc284d3bd9" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38533 } }
I20260621 07:52:22.398268 21063 leader_election.cc:302] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae7bfe939f7348518939eddc284d3bd9; no voters:
I20260621 07:52:22.398703 21063 leader_election.cc:288] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 07:52:22.398862 21067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 07:52:22.399150 21067 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [term 2 LEADER]: Becoming Leader. State: Replica: ae7bfe939f7348518939eddc284d3bd9, State: Running, Role: LEADER
I20260621 07:52:22.399472 21067 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7bfe939f7348518939eddc284d3bd9" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38533 } }
I20260621 07:52:22.400005 21063 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:52:22.400975 21068 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ae7bfe939f7348518939eddc284d3bd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7bfe939f7348518939eddc284d3bd9" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38533 } } }
I20260621 07:52:22.401113 21068 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:22.401463 21076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:52:22.401461 21069 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae7bfe939f7348518939eddc284d3bd9. Latest consensus state: current_term: 2 leader_uuid: "ae7bfe939f7348518939eddc284d3bd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7bfe939f7348518939eddc284d3bd9" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38533 } } }
I20260621 07:52:22.401580 21069 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9 [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:22.404716 21076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:52:22.405581 21076 catalog_manager.cc:1259] Loaded cluster ID: a4e0d8ccbc0b44daa580ababcabc01a6
I20260621 07:52:22.405643 21076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:52:22.407999 21076 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:52:22.408653 21076 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ae7bfe939f7348518939eddc284d3bd9: Loaded TSK: 0
I20260621 07:52:22.409871 21076 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:52:22.483285 21065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:22.483606 21065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:22.483682 21065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:22.487893 21065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:22.487986 21065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:22.488080 21065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:52:22.493782 21065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.1.keytab
--principal=oryx/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:40829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=40811
--tserver_master_addrs=127.12.12.62:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:22.495164 21065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:22.496206 21065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:22.507084 21089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:22.507139 21090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:22.507431 21065 server_base.cc:1034] running on GCE node
W20260621 07:52:22.507700 21092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:22.508206 21065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:22.509119 21065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:22.510357 21065 hybrid_clock.cc:648] HybridClock initialized: now 1782028342510311 us; error 56 us; skew 500 ppm
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028342, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:22.514479 21065 init.cc:377] Logged in from keytab as oryx/127.12.12.1@KRBTEST.COM (short username oryx)
I20260621 07:52:22.515830 21065 webserver.cc:460] Webserver started at http://127.12.12.1:40811/ using document root <none> and password file <none>
I20260621 07:52:22.516644 21065 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:22.516764 21065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:22.522513 21065 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 07:52:22.525557 21099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:22.527252 21065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 07:52:22.527384 21065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "30ca3272b3f843c886227d5dfcc4bc8b"
format_stamp: "Formatted at 2026-06-21 07:52:21 on dist-test-slave-lbpt"
server_key: "deeb6882b1b32ed5770afc7e0a8d7ba8"
server_key_iv: "8d9a56a06956dbf7e0cba2e56b9c9bea"
server_key_version: "encryptionkey@0"
I20260621 07:52:22.528100 21065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:22.543921 21065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:22.547979 21065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:22.548175 21065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:22.548835 21065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:22.549815 21065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:22.549880 21065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:22.549929 21065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:22.549952 21065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:22.560509 21065 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:40829
I20260621 07:52:22.560575 21212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:40829 every 8 connection(s)
I20260621 07:52:22.561640 21065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:52:22.569195 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 21065
I20260621 07:52:22.569713 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:39205
--local_ip_for_outbound_sockets=127.12.12.2
--tserver_master_addrs=127.12.12.62:38533
--webserver_port=36699
--webserver_interface=127.12.12.2
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.12.2
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:22.575034 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:22.563930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:57867)
0621 07:52:22.564741 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:22.564793 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:22.565591 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:22.566688 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:22.566695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:22.567053 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:22.567531 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.567545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.568830 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.568833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.569874 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.569890 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.570764 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.570768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.570951 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.571601 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:22.571617 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:22.573989 (+ 2372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":172,"threads_started":1}
W20260621 07:52:22.575028 21215 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:22.563839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:57867 (local address 127.12.12.62:38533)
0621 07:52:22.564276 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:22.564297 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:22.564948 (+ 651us) server_negotiation.cc:407] Connection header received
0621 07:52:22.565953 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:22.565965 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:22.566256 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:22.566547 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:22.567726 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.568691 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.570033 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.570595 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.570774 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.574212 (+ 3438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:57867: BlockingRecv error: recv got EOF from 127.12.12.1:57867 (error 108)
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
W20260621 07:52:22.575506 21213 heartbeater.cc:644] Failed to heartbeat to 127.12.12.62:38533 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.12.62:38533: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:22.581908 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:22.575955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:45887)
0621 07:52:22.576248 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:22.576262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:22.576326 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:22.576638 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:22.576640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:22.576772 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:22.576874 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.576880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.578043 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.578046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.578712 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.578722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.579411 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.579414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.579486 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.580039 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:22.580051 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:22.581800 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":158}
W20260621 07:52:22.582086 21215 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:22.576074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:45887 (local address 127.12.12.62:38533)
0621 07:52:22.576383 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:22.576390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:22.576407 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:52:22.576455 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:22.576460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:22.576523 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:22.576624 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:22.576995 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.577908 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.578838 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.579253 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.579396 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.581972 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:45887: BlockingRecv error: recv got EOF from 127.12.12.1:45887 (error 108)
Metrics: {"server-negotiator.queue_time_us":201}
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:22.587816 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:22.582491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:33611)
0621 07:52:22.582601 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:22.582618 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:22.582702 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:22.582932 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:22.582934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:22.583055 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:22.583123 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.583127 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.584133 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.584135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.584682 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.584690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.585226 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.585229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.585318 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.585930 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:22.585950 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:22.587708 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260621 07:52:22.587977 21215 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:22.582594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:33611 (local address 127.12.12.62:38533)
0621 07:52:22.582733 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:22.582737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:22.582751 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:52:22.582786 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:22.582788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:22.582830 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:22.582910 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:22.583225 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.584027 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.584791 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.585121 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.585216 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.587865 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:33611: BlockingRecv error: recv got EOF from 127.12.12.1:33611 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260621 07:52:22.588035 21213 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:52:22.685618 21217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:22.685915 21217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:22.685962 21217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:22.690029 21217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:22.690110 21217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:22.690172 21217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:52:22.695933 21217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.2.keytab
--principal=oryx/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:39205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=36699
--tserver_master_addrs=127.12.12.62:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:22.697033 21217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:22.698036 21217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:22.708830 21225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:22.708987 21217 server_base.cc:1034] running on GCE node
W20260621 07:52:22.708830 21223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:22.709158 21222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:22.709754 21217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:22.710572 21217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:22.711747 21217 hybrid_clock.cc:648] HybridClock initialized: now 1782028342711726 us; error 35 us; skew 500 ppm
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028342, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:22.714666 21217 init.cc:377] Logged in from keytab as oryx/127.12.12.2@KRBTEST.COM (short username oryx)
I20260621 07:52:22.715567 21217 webserver.cc:460] Webserver started at http://127.12.12.2:36699/ using document root <none> and password file <none>
I20260621 07:52:22.716156 21217 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:22.716231 21217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:22.720151 21217 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260621 07:52:22.722296 21232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:22.723805 21217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 07:52:22.723942 21217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "06e87cb9f19f4a718949e14729646f59"
format_stamp: "Formatted at 2026-06-21 07:52:21 on dist-test-slave-lbpt"
server_key: "00b92dcbdba1643470b9a99f7d923086"
server_key_iv: "e6766b0bc6a5fd1f842e1b22ec57dde6"
server_key_version: "encryptionkey@0"
I20260621 07:52:22.724574 21217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:22.735343 21217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:22.738531 21217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:22.738734 21217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:22.739336 21217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:22.740170 21217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:22.740245 21217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:22.740311 21217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:22.740370 21217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:22.749874 21217 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:39205
I20260621 07:52:22.749897 21345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:39205 every 8 connection(s)
I20260621 07:52:22.751030 21217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:52:22.756683 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 21217
I20260621 07:52:22.757182 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:33441
--local_ip_for_outbound_sockets=127.12.12.3
--tserver_master_addrs=127.12.12.62:38533
--webserver_port=44455
--webserver_interface=127.12.12.3
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.12.3
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 07:52:22.763506 21215 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:22.753155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:32995 (local address 127.12.12.62:38533)
0621 07:52:22.753430 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:22.753435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:22.754581 (+ 1146us) server_negotiation.cc:407] Connection header received
0621 07:52:22.755433 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:22.755436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:22.755491 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:22.755582 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:22.756970 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.757675 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.759263 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.759645 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.759775 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.763325 (+ 3550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:32995: BlockingRecv error: recv got EOF from 127.12.12.2:32995 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260621 07:52:22.764005 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:22.753672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:32995)
0621 07:52:22.754342 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:22.754392 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:22.755217 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:22.755750 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:22.755756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:22.756116 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:22.756800 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.756823 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.757822 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.757829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.759132 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.759142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.759808 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.759811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.759950 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.760587 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:22.760603 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:22.763108 (+ 2505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":133,"threads_started":1}
W20260621 07:52:22.764652 21346 heartbeater.cc:644] Failed to heartbeat to 127.12.12.62:38533 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.12.62:38533: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:22.770401 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:22.765154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:58537)
0621 07:52:22.765273 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:22.765289 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:22.765378 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:22.765623 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:22.765625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:22.765790 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:22.765878 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.765882 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.766671 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.766677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.767221 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.767228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.767679 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.767683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.767771 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.768359 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:22.768371 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:22.770224 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 07:52:22.770521 21215 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:22.765265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:58537 (local address 127.12.12.62:38533)
0621 07:52:22.765379 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:22.765383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:22.765397 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:52:22.765469 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:22.765472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:22.765516 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:22.765607 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:22.765992 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.766567 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.767345 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.767566 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.767681 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.770372 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:58537: BlockingRecv error: recv got EOF from 127.12.12.2:58537 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:22.776657 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:22.771214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:49803)
0621 07:52:22.771452 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:22.771464 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:22.771526 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:22.771765 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:22.771767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:22.771892 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:22.771964 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.771968 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.772968 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.772970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.773485 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.773493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.774125 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.774127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.774185 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.774704 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:22.774716 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:22.776518 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142}
W20260621 07:52:22.776897 21215 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:22.771364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:49803 (local address 127.12.12.62:38533)
0621 07:52:22.771473 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:22.771477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:22.771493 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:52:22.771612 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:22.771615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:22.771658 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:22.771741 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:22.772062 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.772861 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.773598 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.774006 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.774115 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.776689 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:49803: BlockingRecv error: recv got EOF from 127.12.12.2:49803 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260621 07:52:22.777145 21346 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:52:22.873926 21349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:22.874224 21349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:22.874297 21349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:22.878337 21349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:22.878468 21349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:22.878574 21349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:52:22.884115 21349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/krb5kdc/oryx_127.12.12.3.keytab
--principal=oryx/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:33441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=44455
--tserver_master_addrs=127.12.12.62:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:22.885236 21349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:22.886322 21349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:22.896440 21357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:22.896435 21354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:22.896435 21355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:22.897352 21349 server_base.cc:1034] running on GCE node
I20260621 07:52:22.897735 21349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:22.898341 21349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:22.899631 21349 hybrid_clock.cc:648] HybridClock initialized: now 1782028342899615 us; error 31 us; skew 500 ppm
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028342, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:22.902781 21349 init.cc:377] Logged in from keytab as oryx/127.12.12.3@KRBTEST.COM (short username oryx)
I20260621 07:52:22.903769 21349 webserver.cc:460] Webserver started at http://127.12.12.3:44455/ using document root <none> and password file <none>
I20260621 07:52:22.904345 21349 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:22.904464 21349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:22.908497 21349 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:52:22.910759 21364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:22.912083 21349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:52:22.912184 21349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "9a7523a3fbd647f2bab7f770ad71221d"
format_stamp: "Formatted at 2026-06-21 07:52:21 on dist-test-slave-lbpt"
server_key: "546e6442d80a6aaa0272799250b31b0d"
server_key_iv: "36cde8f0ee63e7a098298c91f74027df"
server_key_version: "encryptionkey@0"
I20260621 07:52:22.912707 21349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:22.926033 21349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:22.929164 21349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:22.929316 21349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:22.929829 21349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:22.930671 21349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:22.930719 21349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:22.930757 21349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:22.930773 21349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:22.939802 21349 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:33441
I20260621 07:52:22.939821 21477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:33441 every 8 connection(s)
I20260621 07:52:22.940789 21349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:52:22.944422 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 21349
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 07:52:22.951870 21215 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:22.942826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:38129 (local address 127.12.12.62:38533)
0621 07:52:22.942968 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:22.942972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:22.943975 (+ 1003us) server_negotiation.cc:407] Connection header received
0621 07:52:22.944956 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:22.944960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:22.945016 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:22.945120 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:22.946395 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.946946 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.948056 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.948276 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.948408 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.951751 (+ 3343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:38129: BlockingRecv error: recv got EOF from 127.12.12.3:38129 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:52:22.952497 21480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:22.943179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:38129)
0621 07:52:22.943813 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:22.943852 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:22.944724 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:22.945284 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:22.945291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:22.945697 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:22.946251 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.946263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.947076 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.947080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.947925 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.947934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.948442 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.948446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.948604 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.949159 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:22.949175 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:22.951655 (+ 2480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":117,"threads_started":1}
W20260621 07:52:22.952950 21478 heartbeater.cc:644] Failed to heartbeat to 127.12.12.62:38533 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.12.62:38533: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
I20260621 07:52:22.954473 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:22.945448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.12.12.62:38533)
0621 07:52:22.945705 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:22.945710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:22.945723 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:52:22.945762 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:22.945765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:22.945804 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:22.945898 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:22.946847 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.947400 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.948274 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.948511 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.948625 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.950086 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:22.950093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:22.950114 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:22.950139 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:22.951970 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:22.952474 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:22.952481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:22.952487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:22.952686 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:22.953009 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:22.953014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:22.953015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:22.953460 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:22.953653 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:22.953947 (+ 294us) server_negotiation.cc:299] Negotiation successful
0621 07:52:22.954292 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":93,"threads_started":1}
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:22.959106 21480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:22.953398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:60281)
0621 07:52:22.953761 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:22.953776 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:22.953851 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:22.954096 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:22.954100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:22.954237 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:22.954322 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.954327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.955107 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.955111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.955761 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.955771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.956294 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.956298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.956398 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.956921 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:22.956936 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:22.958955 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278}
W20260621 07:52:22.959460 21215 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:22.953494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:60281 (local address 127.12.12.62:38533)
0621 07:52:22.953609 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:22.953613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:22.953835 (+ 222us) server_negotiation.cc:407] Connection header received
0621 07:52:22.953931 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:22.953933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:22.953972 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:22.954051 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:22.954458 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.955000 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.955878 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.956174 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.956279 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.959152 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:60281: BlockingRecv error: recv got EOF from 127.12.12.3:60281 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:22 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:22.964586 21480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:22.959646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:60781)
0621 07:52:22.959758 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:22.959771 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:22.959878 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:22.960182 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:22.960184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:22.960311 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:22.960389 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.960393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.961070 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.961073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.961599 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:22.961606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.962053 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.962055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:22.962113 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.962813 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:22.962827 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:22.964494 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 07:52:22.964740 21215 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:22.959871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:60781 (local address 127.12.12.62:38533)
0621 07:52:22.960011 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:22.960014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:22.960025 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:52:22.960054 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:22.960056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:22.960087 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:22.960150 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:22.960492 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.960972 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.961739 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:22.961945 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:22.962032 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:22.964618 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:60781: BlockingRecv error: recv got EOF from 127.12.12.3:60781 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260621 07:52:22.964816 21478 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 21 07:52:23 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:23 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:23.594650 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:23.588754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:49849)
0621 07:52:23.589036 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:23.589050 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:23.589139 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:23.589389 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:23.589392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:23.589571 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:23.589696 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:23.589702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:23.590567 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:23.590570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:23.591158 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:23.591166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:23.591703 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:23.591706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:23.591789 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:23.592413 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:23.592425 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:23.594518 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":111,"threads_started":1}
W20260621 07:52:23.594870 21484 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:23.588873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:49849 (local address 127.12.12.62:38533)
0621 07:52:23.589107 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:23.589111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:23.589124 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:52:23.589222 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:23.589225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:23.589268 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:23.589337 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:23.589808 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:23.590380 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:23.591296 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:23.591561 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:23.591652 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:23.594686 (+ 3034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:49849: BlockingRecv error: recv got EOF from 127.12.12.1:49849 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
Jun 21 07:52:23 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:23 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:23.784641 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:23.778075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:33059)
0621 07:52:23.778398 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:23.778443 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:23.778568 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:23.778954 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:23.778958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:23.779225 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:23.779401 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:23.779412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:23.780529 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:23.780533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:23.781128 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:23.781138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:23.781646 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:23.781650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:23.781742 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:23.782322 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:23.782335 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:23.784454 (+ 2119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":102,"threads_started":1}
W20260621 07:52:23.784704 21484 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:23.778207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:33059 (local address 127.12.12.62:38533)
0621 07:52:23.778400 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:23.778404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:23.778534 (+ 130us) server_negotiation.cc:407] Connection header received
0621 07:52:23.778654 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:23.778658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:23.778712 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:23.778823 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:23.779570 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:23.780407 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:23.781265 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:23.781518 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:23.781637 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:23.784582 (+ 2945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:33059: BlockingRecv error: recv got EOF from 127.12.12.2:33059 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 21 07:52:23 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:23 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:23.971248 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:23.965461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:41001)
0621 07:52:23.965736 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:23.965750 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:23.965860 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:23.966109 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:23.966111 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:23.966289 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:23.966395 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:23.966401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:23.967188 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:23.967191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:23.967761 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:23.967769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:23.968230 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:23.968234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:23.968302 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:23.968903 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:23.968916 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:23.971061 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":100,"threads_started":1}
W20260621 07:52:23.971386 21484 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:23.965597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:41001 (local address 127.12.12.62:38533)
0621 07:52:23.965743 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:23.965746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:23.965826 (+ 80us) server_negotiation.cc:407] Connection header received
0621 07:52:23.965943 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:23.965945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:23.965991 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:23.966067 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:23.966515 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:23.967077 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:23.967877 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:23.968099 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:23.968186 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:23.971242 (+ 3056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:41001: BlockingRecv error: recv got EOF from 127.12.12.3:41001 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 21 07:52:24 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:24 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:24.602631 21487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:24.595624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:48797)
0621 07:52:24.596025 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:24.596045 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:24.596169 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:24.596473 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:24.596476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:24.596735 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:24.596849 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:24.596855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:24.597900 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:24.597904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:24.598578 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:24.598608 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:24.599151 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:24.599155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:24.599254 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:24.599875 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:24.599887 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:24.602418 (+ 2531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":123,"threads_started":1}
W20260621 07:52:24.602746 21488 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:24.595882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:48797 (local address 127.12.12.62:38533)
0621 07:52:24.596160 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:24.596164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:24.596179 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:52:24.596248 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:24.596250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:24.596293 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:24.596388 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:24.596997 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:24.597748 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:24.598736 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:24.599009 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:24.599149 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:24.602599 (+ 3450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:48797: BlockingRecv error: recv got EOF from 127.12.12.1:48797 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":102,"threads_started":1}
Jun 21 07:52:24 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:24 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:24.791740 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:24.785669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:50451)
0621 07:52:24.785992 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:24.786006 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:24.786108 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:24.786363 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:24.786365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:24.786570 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:24.786671 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:24.786676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:24.787453 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:24.787457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:24.788077 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:24.788086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:24.788705 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:24.788709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:24.788824 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:24.789427 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:24.789439 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:24.791545 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":96,"threads_started":1}
W20260621 07:52:24.791803 21488 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:24.785774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:50451 (local address 127.12.12.62:38533)
0621 07:52:24.785981 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:24.785985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:24.786060 (+ 75us) server_negotiation.cc:407] Connection header received
0621 07:52:24.786195 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:24.786198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:24.786240 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:24.786319 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:24.786779 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:24.787312 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:24.788219 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:24.788575 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:24.788701 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:24.791686 (+ 2985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:50451: BlockingRecv error: recv got EOF from 127.12.12.2:50451 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 21 07:52:24 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:24 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:24.977759 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:24.972310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:49259)
0621 07:52:24.972641 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:24.972655 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:24.972746 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:24.972994 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:24.972996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:24.973176 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:24.973276 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:24.973281 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:24.974032 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:24.974034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:24.974655 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:24.974665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:24.975102 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:24.975104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:24.975162 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:24.975690 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:24.975702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:24.977618 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":153,"threads_started":1}
W20260621 07:52:24.977905 21488 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:24.972614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:49259 (local address 127.12.12.62:38533)
0621 07:52:24.972769 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:24.972773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:24.972787 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:52:24.972826 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:24.972828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:24.972878 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:24.972953 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:24.973396 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:24.973929 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:24.974762 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:24.975001 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:24.975084 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:24.977757 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:49259: BlockingRecv error: recv got EOF from 127.12.12.3:49259 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 21 07:52:25 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:25 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:25.609804 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:25.603598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:37677)
0621 07:52:25.603904 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:25.603925 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:25.604054 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:25.604418 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:25.604422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:25.604651 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:25.604755 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:25.604761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:25.605805 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:25.605808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:25.606366 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:25.606374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:25.607042 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:25.607045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:25.607113 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:25.607729 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:25.607741 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:25.609683 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":86,"threads_started":1}
W20260621 07:52:25.610044 21492 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:25.603677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:37677 (local address 127.12.12.62:38533)
0621 07:52:25.603985 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:25.603989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:25.604003 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:52:25.604173 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:25.604177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:25.604232 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:25.604340 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:25.604926 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:25.605677 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:25.606673 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:25.606905 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:25.606998 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:25.609870 (+ 2872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:37677: BlockingRecv error: recv got EOF from 127.12.12.1:37677 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":118,"threads_started":1}
Jun 21 07:52:25 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:25 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:25.799141 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:25.792699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:34991)
0621 07:52:25.793078 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:25.793099 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:25.793245 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:25.793585 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:25.793589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:25.793797 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:25.793890 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:25.793896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:25.794773 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:25.794776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:25.795346 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:25.795354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:25.795854 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:25.795857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:25.795942 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:25.796629 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:25.796641 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:25.798939 (+ 2298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":90,"threads_started":1}
W20260621 07:52:25.799209 21492 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:25.792853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:34991 (local address 127.12.12.62:38533)
0621 07:52:25.793045 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:25.793049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:25.793181 (+ 132us) server_negotiation.cc:407] Connection header received
0621 07:52:25.793397 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:25.793401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:25.793460 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:25.793563 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:25.794017 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:25.794647 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:25.795481 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:25.795720 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:25.795852 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:25.799096 (+ 3244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:34991: BlockingRecv error: recv got EOF from 127.12.12.2:34991 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 21 07:52:25 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:25 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:25.984938 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:25.978796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:58313)
0621 07:52:25.979131 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:25.979145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:25.979252 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:25.979527 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:25.979529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:25.979722 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:25.979824 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:25.979829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:25.980611 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:25.980614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:25.981200 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:25.981209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:25.981736 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:25.981739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:25.981845 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:25.982544 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:25.982556 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:25.984741 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":145,"threads_started":1}
W20260621 07:52:25.984997 21492 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:25.978971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:58313 (local address 127.12.12.62:38533)
0621 07:52:25.979121 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:25.979125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:25.979225 (+ 100us) server_negotiation.cc:407] Connection header received
0621 07:52:25.979340 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:25.979342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:25.979393 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:25.979484 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:25.979955 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:25.980497 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:25.981349 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:25.981578 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:25.981708 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:25.984883 (+ 3175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:58313: BlockingRecv error: recv got EOF from 127.12.12.3:58313 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 21 07:52:26 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:26 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:26.620381 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:26.610662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:43267)
0621 07:52:26.610922 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:26.610940 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:26.611054 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:26.611288 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:26.611290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:26.611520 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:26.611617 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:26.611623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:26.612403 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:26.612406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:26.613012 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:26.613021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:26.613468 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:26.613471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:26.613535 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:26.618122 (+ 4587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:26.618136 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:26.620212 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
W20260621 07:52:26.620563 21496 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:26.610775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:43267 (local address 127.12.12.62:38533)
0621 07:52:26.611101 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:26.611105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:26.611117 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:52:26.611151 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:26.611153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:26.611194 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:26.611275 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:26.611741 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:26.612275 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:26.613119 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:26.613337 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:26.613441 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:26.620401 (+ 6960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:43267: BlockingRecv error: recv got EOF from 127.12.12.1:43267 (error 108)
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
Jun 21 07:52:26 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:26 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:26.806486 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:26.800040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:56037)
0621 07:52:26.800454 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:26.800479 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:26.800582 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:26.800887 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:26.800890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:26.801126 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:26.801294 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:26.801300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:26.802383 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:26.802387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:26.803035 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:26.803047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:26.803563 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:26.803567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:26.803661 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:26.804278 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:26.804290 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:26.806271 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":127,"threads_started":1}
W20260621 07:52:26.806571 21496 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:26.800161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:56037 (local address 127.12.12.62:38533)
0621 07:52:26.800324 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:26.800329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:26.800555 (+ 226us) server_negotiation.cc:407] Connection header received
0621 07:52:26.800667 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:26.800669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:26.800741 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:26.800837 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:26.801487 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:26.802277 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:26.803172 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:26.803426 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:26.803546 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:26.806444 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:56037: BlockingRecv error: recv got EOF from 127.12.12.2:56037 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 21 07:52:26 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:26 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:26.991820 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:26.985820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:33839)
0621 07:52:26.986130 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:26.986143 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:26.986252 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:26.986593 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:26.986596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:26.986786 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:26.986888 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:26.986894 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:26.987725 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:26.987728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:26.988360 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:26.988369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:26.988822 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:26.988825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:26.988894 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:26.989593 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:26.989606 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:26.991694 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":135,"threads_started":1}
W20260621 07:52:26.991968 21496 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:26.985971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:33839 (local address 127.12.12.62:38533)
0621 07:52:26.986115 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:26.986118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:26.986203 (+ 85us) server_negotiation.cc:407] Connection header received
0621 07:52:26.986334 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:26.986336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:26.986378 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:26.986510 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:26.987064 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:26.987598 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:26.988473 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:26.988707 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:26.988786 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:26.991838 (+ 3052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:33839: BlockingRecv error: recv got EOF from 127.12.12.3:33839 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 21 07:52:27 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:27 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:27.627350 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:27.621329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:41959)
0621 07:52:27.621672 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:27.621686 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:27.621781 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:27.622018 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:27.622020 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:27.622201 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:27.622303 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:27.622308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:27.623218 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:27.623224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:27.623806 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:27.623815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:27.624325 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:27.624329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:27.624411 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:27.624962 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:27.624974 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:27.627158 (+ 2184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":124,"threads_started":1}
W20260621 07:52:27.627472 21500 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:27.621447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:41959 (local address 127.12.12.62:38533)
0621 07:52:27.621705 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:27.621709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:27.621737 (+ 28us) server_negotiation.cc:407] Connection header received
0621 07:52:27.621846 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:27.621848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:27.621891 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:27.621979 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:27.622480 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:27.623091 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:27.623952 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:27.624202 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:27.624322 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:27.627333 (+ 3011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:41959: BlockingRecv error: recv got EOF from 127.12.12.1:41959 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":64,"threads_started":1}
Jun 21 07:52:27 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:27 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:27.813491 21501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:27.807487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:38723)
0621 07:52:27.807895 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:27.807915 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:27.808026 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:27.808475 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:27.808478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:27.808771 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:27.808877 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:27.808884 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:27.809622 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:27.809624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:27.810199 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:27.810206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:27.810721 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:27.810724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:27.810801 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:27.811319 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:27.811331 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:27.813367 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
W20260621 07:52:27.813809 21500 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:27.807634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:38723 (local address 127.12.12.62:38533)
0621 07:52:27.807799 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:27.807804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:27.808007 (+ 203us) server_negotiation.cc:407] Connection header received
0621 07:52:27.808142 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:27.808147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:27.808202 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:27.808287 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:27.808990 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:27.809520 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:27.810318 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:27.810586 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:27.810670 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:27.813580 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:38723: BlockingRecv error: recv got EOF from 127.12.12.2:38723 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 21 07:52:27 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:27 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:28.000699 21502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:27.992865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:39695)
0621 07:52:27.993172 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:27.993187 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:27.993337 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:27.993629 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:27.993632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:27.993853 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:27.993950 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:27.993955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:27.994888 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:27.994894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:27.996059 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:27.996073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:27.996868 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:27.996872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:27.996967 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:27.998024 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:27.998040 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:28.000537 (+ 2497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
W20260621 07:52:28.001009 21500 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:27.992996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:39695 (local address 127.12.12.62:38533)
0621 07:52:27.993136 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:27.993142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:27.993285 (+ 143us) server_negotiation.cc:407] Connection header received
0621 07:52:27.993454 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:27.993457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:27.993514 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:27.993610 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:27.994083 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:27.994723 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:27.996226 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:27.996679 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:27.996828 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:28.000666 (+ 3838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:39695: BlockingRecv error: recv got EOF from 127.12.12.3:39695 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 21 07:52:28 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:28 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:28.634598 21503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:28.628290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:46801)
0621 07:52:28.628628 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:28.628643 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:28.628741 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:28.629054 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:28.629056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:28.629266 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:28.629413 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:28.629419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:28.630262 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:28.630265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:28.631009 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:28.631019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:28.631531 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:28.631534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:28.631601 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:28.632230 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:28.632242 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:28.634457 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":118,"threads_started":1}
W20260621 07:52:28.634784 21504 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:28.628471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:46801 (local address 127.12.12.62:38533)
0621 07:52:28.628689 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:28.628693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:28.628707 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:52:28.628836 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:28.628839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:28.628921 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:28.629008 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:28.629555 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:28.630130 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:28.631130 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:28.631368 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:28.631468 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:28.634614 (+ 3146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:46801: BlockingRecv error: recv got EOF from 127.12.12.1:46801 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":60,"threads_started":1}
Jun 21 07:52:28 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:28 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:28.819993 21505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:28.814399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:45817)
0621 07:52:28.814765 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:28.814777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:28.814884 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:28.815181 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:28.815184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:28.815397 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:28.815492 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:28.815498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:28.816316 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:28.816319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:28.816890 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:28.816899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:28.817401 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:28.817405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:28.817505 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:28.818006 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:28.818016 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:28.819873 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":92,"threads_started":1}
W20260621 07:52:28.820144 21504 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:28.814465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:45817 (local address 127.12.12.62:38533)
0621 07:52:28.814614 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:28.814619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:28.814852 (+ 233us) server_negotiation.cc:407] Connection header received
0621 07:52:28.815008 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:28.815011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:28.815069 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:28.815158 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:28.815616 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:28.816171 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:28.817019 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:28.817237 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:28.817346 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:28.820018 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:45817: BlockingRecv error: recv got EOF from 127.12.12.2:45817 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 21 07:52:29 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:29 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:29.012079 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:29.001565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:46045)
0621 07:52:29.001907 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:29.001924 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:29.002056 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:29.002464 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:29.002468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:29.002731 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:29.002882 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:29.002889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:29.005290 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:29.005293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:29.006370 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:29.006381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:29.008480 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:29.008487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:29.008602 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:29.009257 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:29.009278 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:29.011894 (+ 2616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":80,"threads_started":1}
W20260621 07:52:29.012133 21504 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:29.001946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:46045 (local address 127.12.12.62:38533)
0621 07:52:29.002077 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:29.002081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:29.002100 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:52:29.002150 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:29.002155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:29.002221 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:29.004375 (+ 2154us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:29.004527 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:29.005053 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:29.008022 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:29.008329 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:29.008437 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:29.012028 (+ 3591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:46045: BlockingRecv error: recv got EOF from 127.12.12.3:46045 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 21 07:52:29 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:29 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:29.640779 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:29.635372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:34593)
0621 07:52:29.635722 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:29.635735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:29.635833 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:29.636064 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:29.636066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:29.636250 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:29.636347 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:29.636352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:29.637097 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:29.637100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:29.637670 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:29.637677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:29.638112 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:29.638115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:29.638178 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:29.638741 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:29.638752 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:29.640660 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":92,"threads_started":1}
W20260621 07:52:29.640951 21507 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:29.635422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:34593 (local address 127.12.12.62:38533)
0621 07:52:29.635690 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:29.635693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:29.635808 (+ 115us) server_negotiation.cc:407] Connection header received
0621 07:52:29.635900 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:29.635902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:29.635945 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:29.636027 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:29.636449 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:29.636997 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:29.637781 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:29.638008 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:29.638088 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:29.640801 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:34593: BlockingRecv error: recv got EOF from 127.12.12.1:34593 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":109,"threads_started":1}
Jun 21 07:52:29 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:29 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:29.827833 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:29.820976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:48391)
0621 07:52:29.821411 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:29.821468 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:29.821586 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:29.821908 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:29.821911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:29.822100 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:29.822194 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:29.822199 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:29.823121 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:29.823124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:29.823881 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:29.823892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:29.824507 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:29.824514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:29.824605 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:29.825316 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:29.825334 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:29.827659 (+ 2325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":102,"threads_started":1}
W20260621 07:52:29.827900 21507 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:29.821072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:48391 (local address 127.12.12.62:38533)
0621 07:52:29.821253 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:29.821257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:29.821581 (+ 324us) server_negotiation.cc:407] Connection header received
0621 07:52:29.821716 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:29.821719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:29.821768 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:29.821848 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:29.822311 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:29.822977 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:29.824012 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:29.824313 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:29.824437 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:29.827790 (+ 3353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:48391: BlockingRecv error: recv got EOF from 127.12.12.2:48391 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 21 07:52:30 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:30 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:30.019279 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:30.012992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:54825)
0621 07:52:30.013312 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:30.013326 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:30.013496 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:30.013748 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:30.013750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:30.013938 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:30.014039 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:30.014044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:30.014897 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:30.014901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:30.015455 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:30.015490 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:30.016036 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:30.016039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:30.016150 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:30.016893 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:30.016909 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:30.019100 (+ 2191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":75,"threads_started":1}
W20260621 07:52:30.019340 21507 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:30.013113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:54825 (local address 127.12.12.62:38533)
0621 07:52:30.013329 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:30.013333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:30.013438 (+ 105us) server_negotiation.cc:407] Connection header received
0621 07:52:30.013586 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:30.013589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:30.013636 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:30.013710 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:30.014144 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:30.014782 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:30.015606 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:30.015912 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:30.016030 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:30.019231 (+ 3201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:54825: BlockingRecv error: recv got EOF from 127.12.12.3:54825 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Jun 21 07:52:30 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:30 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:30.647423 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:30.641671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:47839)
0621 07:52:30.641938 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:30.641951 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:30.642042 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:30.642314 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:30.642316 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:30.642586 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:30.642685 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:30.642691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:30.643496 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:30.643500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:30.644053 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:30.644061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:30.644609 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:30.644612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:30.644712 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:30.645211 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:30.645223 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:30.647295 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":95,"threads_started":1}
W20260621 07:52:30.647646 21512 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:30.641867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:47839 (local address 127.12.12.62:38533)
0621 07:52:30.642105 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:30.642108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:30.642120 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:52:30.642154 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:30.642156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:30.642196 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:30.642275 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:30.642785 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:30.643327 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:30.644187 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:30.644458 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:30.644549 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:30.647490 (+ 2941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:47839: BlockingRecv error: recv got EOF from 127.12.12.1:47839 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
Jun 21 07:52:30 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:30 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:30.835395 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:30.828833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:45903)
0621 07:52:30.829104 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:30.829118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:30.829266 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:30.829596 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:30.829599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:30.829875 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:30.829982 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:30.829989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:30.831199 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:30.831203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:30.831793 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:30.831802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:30.832298 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:30.832301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:30.832368 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:30.832924 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:30.832939 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:30.835268 (+ 2329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":70,"threads_started":1}
W20260621 07:52:30.835645 21512 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:30.828960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:45903 (local address 127.12.12.62:38533)
0621 07:52:30.829142 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:30.829146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:30.829248 (+ 102us) server_negotiation.cc:407] Connection header received
0621 07:52:30.829395 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:30.829399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:30.829455 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:30.829559 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:30.830111 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:30.831075 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:30.831928 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:30.832156 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:30.832286 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:30.835484 (+ 3198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:45903: BlockingRecv error: recv got EOF from 127.12.12.2:45903 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 21 07:52:31 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:31 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:31.026389 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:31.020224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:42557)
0621 07:52:31.020538 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:31.020551 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:31.020653 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:31.020910 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:31.020912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:31.021102 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:31.021200 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:31.021206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:31.022008 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:31.022011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:31.022643 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:31.022652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:31.023120 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:31.023123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:31.023191 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:31.023767 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:31.023779 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:31.026218 (+ 2439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
W20260621 07:52:31.026491 21512 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:31.020357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:42557 (local address 127.12.12.62:38533)
0621 07:52:31.020500 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:31.020504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:31.020632 (+ 128us) server_negotiation.cc:407] Connection header received
0621 07:52:31.020733 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:31.020735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:31.020782 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:31.020862 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:31.021313 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:31.021877 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:31.022772 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:31.022993 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:31.023116 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:31.026342 (+ 3226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:42557: BlockingRecv error: recv got EOF from 127.12.12.3:42557 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 21 07:52:31 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:31 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:31.658855 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:31.648329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:53219)
0621 07:52:31.648639 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:31.648652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:31.648748 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:31.649059 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:31.649062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:31.649248 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:31.649356 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:31.649368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:31.650135 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:31.650137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:31.650789 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:31.650798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:31.651355 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:31.651365 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:31.651474 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:31.656406 (+ 4932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:31.656427 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:31.658729 (+ 2302us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":112,"threads_started":1}
W20260621 07:52:31.659087 21516 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:31.648463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:53219 (local address 127.12.12.62:38533)
0621 07:52:31.648715 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:31.648719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:31.648733 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:52:31.648842 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:31.648845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:31.648888 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:31.648976 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:31.649474 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:31.650031 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:31.650906 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:31.651223 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:31.651346 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:31.658903 (+ 7557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:53219: BlockingRecv error: recv got EOF from 127.12.12.1:53219 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":96,"threads_started":1}
Jun 21 07:52:31 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:31 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:31.842545 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:31.836290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:43545)
0621 07:52:31.836645 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:31.836658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:31.836772 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:31.837036 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:31.837038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:31.837221 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:31.837319 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:31.837323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:31.838118 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:31.838121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:31.838772 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:31.838781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:31.839306 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:31.839309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:31.839420 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:31.840059 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:31.840071 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:31.842266 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":117,"threads_started":1}
W20260621 07:52:31.842658 21516 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:31.836354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:43545 (local address 127.12.12.62:38533)
0621 07:52:31.836522 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:31.836528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:31.836761 (+ 233us) server_negotiation.cc:407] Connection header received
0621 07:52:31.836857 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:31.836860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:31.836908 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:31.836996 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:31.837449 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:31.838006 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:31.838930 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:31.839158 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:31.839255 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:31.842525 (+ 3270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:43545: BlockingRecv error: recv got EOF from 127.12.12.2:43545 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 21 07:52:32 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:32 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:32.033450 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:32.027288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:36579)
0621 07:52:32.027635 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:32.027654 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:32.027819 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:32.028187 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:32.028190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:32.028459 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:32.028558 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:32.028564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:32.029349 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:32.029355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:32.029928 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:32.029936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:32.030451 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:32.030454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:32.030540 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:32.031072 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:32.031084 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:32.033292 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":138,"threads_started":1}
W20260621 07:52:32.033574 21516 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:32.027513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:36579 (local address 127.12.12.62:38533)
0621 07:52:32.027650 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:32.027655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:32.027751 (+ 96us) server_negotiation.cc:407] Connection header received
0621 07:52:32.027945 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:32.027953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:32.028001 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:32.028146 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:32.028683 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:32.029227 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:32.030057 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:32.030285 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:32.030448 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:32.033441 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:36579: BlockingRecv error: recv got EOF from 127.12.12.3:36579 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 07:52:32 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:32 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:32.666502 21519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:32.659836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:48671)
0621 07:52:32.660114 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:32.660132 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:32.660233 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:32.660533 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:32.660536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:32.660759 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:32.660857 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:32.660863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:32.661981 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:32.661985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:32.662897 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:32.662908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:32.663527 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:32.663536 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:32.663604 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:32.664198 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:32.664210 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:32.666273 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
W20260621 07:52:32.666646 21520 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:32.659976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:48671 (local address 127.12.12.62:38533)
0621 07:52:32.660270 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:32.660274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:32.660287 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:52:32.660366 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:32.660368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:32.660411 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:32.660498 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:32.660987 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:32.661814 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:32.663030 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:32.663376 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:32.663499 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:32.666506 (+ 3007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:48671: BlockingRecv error: recv got EOF from 127.12.12.1:48671 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
Jun 21 07:52:32 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:32 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:32.849255 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:32.843495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:48147)
0621 07:52:32.843756 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:32.843770 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:32.843903 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:32.844191 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:32.844194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:32.844408 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:32.844499 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:32.844505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:32.845272 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:32.845277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:32.845869 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:32.845878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:32.846487 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:32.846490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:32.846581 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:32.847165 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:32.847177 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:32.849053 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
W20260621 07:52:32.849291 21520 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:32.843548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:48147 (local address 127.12.12.62:38533)
0621 07:52:32.843720 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:32.843725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:32.843857 (+ 132us) server_negotiation.cc:407] Connection header received
0621 07:52:32.844011 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:32.844015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:32.844067 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:32.844164 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:32.844622 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:32.845156 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:32.846100 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:32.846337 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:32.846479 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:32.849186 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:48147: BlockingRecv error: recv got EOF from 127.12.12.2:48147 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 07:52:33 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:33 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:33.040022 21522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:33.034336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:52529)
0621 07:52:33.034741 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:33.034756 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:33.034857 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:33.035081 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:33.035082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:33.035264 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:33.035368 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:33.035373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:33.036114 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:33.036116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:33.036711 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:33.036720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:33.037324 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:33.037327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:33.037427 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:33.037943 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:33.037955 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:33.039894 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":136,"threads_started":1}
W20260621 07:52:33.040208 21520 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:33.034494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:52529 (local address 127.12.12.62:38533)
0621 07:52:33.034653 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:33.034658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:33.034835 (+ 177us) server_negotiation.cc:407] Connection header received
0621 07:52:33.034940 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:33.034942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:33.034999 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:33.035069 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:33.035472 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:33.036009 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:33.036978 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:33.037208 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:33.037310 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:33.040065 (+ 2755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:52529: BlockingRecv error: recv got EOF from 127.12.12.3:52529 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 21 07:52:33 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:33 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:33.672865 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:33.667244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:51329)
0621 07:52:33.667549 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:33.667562 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:33.667654 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:33.667901 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:33.667903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:33.668076 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:33.668172 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:33.668177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:33.668907 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:33.668910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:33.669510 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:33.669518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:33.669996 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:33.669999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:33.670075 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:33.670681 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:33.670694 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:33.672686 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":90,"threads_started":1}
W20260621 07:52:33.672941 21524 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:33.667386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:51329 (local address 127.12.12.62:38533)
0621 07:52:33.667629 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:33.667633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:33.667646 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:52:33.667739 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:33.667741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:33.667785 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:33.667854 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:33.668273 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:33.668803 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:33.669628 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:33.669856 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:33.669942 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:33.672814 (+ 2872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:51329: BlockingRecv error: recv got EOF from 127.12.12.1:51329 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
Jun 21 07:52:33 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:33 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:33.856014 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:33.850227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:49385)
0621 07:52:33.850563 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:33.850577 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:33.850680 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:33.850923 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:33.850925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:33.851101 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:33.851197 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:33.851202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:33.851980 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:33.851983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:33.852568 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:33.852576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:33.853019 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:33.853021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:33.853084 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:33.853617 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:33.853628 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:33.855838 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":125,"threads_started":1}
W20260621 07:52:33.856052 21524 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:33.850318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:49385 (local address 127.12.12.62:38533)
0621 07:52:33.850501 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:33.850505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:33.850678 (+ 173us) server_negotiation.cc:407] Connection header received
0621 07:52:33.850751 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:33.850753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:33.850807 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:33.850881 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:33.851312 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:33.851864 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:33.852678 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:33.852909 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:33.852988 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:33.855951 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:49385: BlockingRecv error: recv got EOF from 127.12.12.2:49385 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 21 07:52:34 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:34 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:34.046974 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:34.040966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:47865)
0621 07:52:34.041263 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:34.041277 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:34.041424 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:34.041755 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:34.041758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:34.041975 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:34.042068 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:34.042074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:34.042914 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:34.042919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:34.043503 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:34.043511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:34.043979 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:34.043983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:34.044060 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:34.044641 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:34.044653 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:34.046779 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
W20260621 07:52:34.047039 21524 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:34.041029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:47865 (local address 127.12.12.62:38533)
0621 07:52:34.041192 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:34.041198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:34.041361 (+ 163us) server_negotiation.cc:407] Connection header received
0621 07:52:34.041576 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:34.041581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:34.041635 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:34.041732 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:34.042196 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:34.042797 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:34.043636 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:34.043857 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:34.043981 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:34.046915 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:47865: BlockingRecv error: recv got EOF from 127.12.12.3:47865 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 21 07:52:34 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:34 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:34.679273 21527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:34.673638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:33255)
0621 07:52:34.673897 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:34.673912 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:34.674015 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:34.674283 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:34.674286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:34.674565 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:34.674669 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:34.674676 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:34.675574 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:34.675577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:34.676134 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:34.676142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:34.676582 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:34.676585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:34.676650 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:34.677242 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:34.677261 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:34.679137 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
W20260621 07:52:34.679430 21528 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:34.673796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:33255 (local address 127.12.12.62:38533)
0621 07:52:34.674055 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:34.674059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:34.674071 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:52:34.674105 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:34.674107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:34.674146 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:34.674238 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:34.674771 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:34.675461 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:34.676237 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:34.676476 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:34.676557 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:34.679286 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:33255: BlockingRecv error: recv got EOF from 127.12.12.1:33255 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
Jun 21 07:52:34 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:34 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:34.862712 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:34.856909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:50983)
0621 07:52:34.857310 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:34.857323 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:34.857468 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:34.857766 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:34.857769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:34.857984 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:34.858078 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:34.858083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:34.858942 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:34.858948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:34.859522 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:34.859530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:34.859992 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:34.859995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:34.860072 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:34.860660 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:34.860672 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:34.862583 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":93,"threads_started":1}
W20260621 07:52:34.862929 21528 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:34.857042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:50983 (local address 127.12.12.62:38533)
0621 07:52:34.857219 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:34.857224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:34.857408 (+ 184us) server_negotiation.cc:407] Connection header received
0621 07:52:34.857594 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:34.857597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:34.857651 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:34.857741 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:34.858207 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:34.858828 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:34.859651 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:34.859875 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:34.859995 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:34.862757 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:50983: BlockingRecv error: recv got EOF from 127.12.12.2:50983 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 21 07:52:35 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:35 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:35.053264 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:35.047873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:59033)
0621 07:52:35.048159 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:35.048173 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:35.048277 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:35.048510 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:35.048513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:35.048694 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:35.048788 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:35.048794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:35.049532 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:35.049535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:35.050090 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:35.050097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:35.050585 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:35.050587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:35.050649 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:35.051128 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:35.051139 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:35.053072 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
W20260621 07:52:35.053357 21528 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:35.048013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:59033 (local address 127.12.12.62:38533)
0621 07:52:35.048183 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:35.048187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:35.048247 (+ 60us) server_negotiation.cc:407] Connection header received
0621 07:52:35.048361 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:35.048363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:35.048408 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:35.048504 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:35.048889 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:35.049417 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:35.050203 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:35.050457 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:35.050570 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:35.053244 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:59033: BlockingRecv error: recv got EOF from 127.12.12.3:59033 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 21 07:52:35 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:35 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:35.685958 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:35.680178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:37977)
0621 07:52:35.680514 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:35.680538 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:35.680673 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:35.680995 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:35.680999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:35.681240 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:35.681367 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:35.681375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:35.682208 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:35.682211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:35.682883 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:35.682893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:35.683339 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:35.683341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:35.683445 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:35.683943 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:35.683955 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:35.685815 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":84,"threads_started":1}
W20260621 07:52:35.686079 21532 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:35.680311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:37977 (local address 127.12.12.62:38533)
0621 07:52:35.680544 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:35.680549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:35.680591 (+ 42us) server_negotiation.cc:407] Connection header received
0621 07:52:35.680782 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:35.680786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:35.680846 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:35.680925 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:35.681544 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:35.682106 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:35.682996 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:35.683218 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:35.683301 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:35.685949 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:37977: BlockingRecv error: recv got EOF from 127.12.12.1:37977 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":65,"threads_started":1}
Jun 21 07:52:35 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:35 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:35.869117 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:35.863586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:40075)
0621 07:52:35.863863 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:35.863879 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:35.863983 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:35.864236 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:35.864238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:35.864415 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:35.864541 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:35.864548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:35.865289 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:35.865291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:35.865856 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:35.865864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:35.866295 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:35.866298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:35.866356 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:35.866962 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:35.866974 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:35.868927 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":103,"threads_started":1}
W20260621 07:52:35.869158 21532 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:35.863793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:40075 (local address 127.12.12.62:38533)
0621 07:52:35.863951 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:35.863954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:35.863968 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:52:35.864073 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:35.864076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:35.864119 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:35.864198 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:35.864654 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:35.865195 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:35.865968 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:35.866194 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:35.866278 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:35.869056 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:40075: BlockingRecv error: recv got EOF from 127.12.12.2:40075 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 21 07:52:36 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:36 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:36.060046 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:36.054120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:53109)
0621 07:52:36.054445 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:36.054459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:36.054555 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:36.054854 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:36.054857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:36.055049 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:36.055168 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:36.055174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:36.055990 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:36.055993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:36.056588 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:36.056597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:36.057164 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:36.057169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:36.057261 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:36.057867 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:36.057879 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:36.059845 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":85,"threads_started":1}
W20260621 07:52:36.060075 21532 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:36.054282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:53109 (local address 127.12.12.62:38533)
0621 07:52:36.054498 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:36.054503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:36.054525 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:52:36.054649 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:36.054651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:36.054699 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:36.054769 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:36.055317 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:36.055866 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:36.056732 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:36.057044 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:36.057159 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:36.059968 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:53109: BlockingRecv error: recv got EOF from 127.12.12.3:53109 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 21 07:52:36 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:36 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:36.697636 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:36.686925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:35887)
0621 07:52:36.687189 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:36.687210 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:36.687336 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:36.687612 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:36.687614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:36.687815 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:36.687912 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:36.687918 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:36.688689 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:36.688692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:36.689276 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:36.689284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:36.689810 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:36.689813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:36.689897 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:36.695436 (+ 5539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:36.695453 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:36.697505 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
W20260621 07:52:36.697918 21536 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:36.687069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:35887 (local address 127.12.12.62:38533)
0621 07:52:36.687399 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:36.687402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:36.687415 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:52:36.687453 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:36.687455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:36.687494 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:36.687572 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:36.688030 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:36.688578 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:36.689418 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:36.689645 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:36.689726 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:36.697781 (+ 8055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:35887: BlockingRecv error: recv got EOF from 127.12.12.1:35887 (error 108)
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":83,"threads_started":1}
Jun 21 07:52:36 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:36 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:36.875825 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:36.870011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:50203)
0621 07:52:36.870513 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:36.870527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:36.870662 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:36.870943 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:36.870946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:36.871170 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:36.871265 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:36.871271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:36.872056 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:36.872061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:36.872647 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:36.872655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:36.873123 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:36.873126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:36.873215 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:36.873797 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:36.873810 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:36.875693 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":118,"threads_started":1}
W20260621 07:52:36.876003 21536 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:36.870116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:50203 (local address 127.12.12.62:38533)
0621 07:52:36.870275 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:36.870279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:36.870610 (+ 331us) server_negotiation.cc:407] Connection header received
0621 07:52:36.870774 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:36.870777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:36.870830 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:36.870917 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:36.871393 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:36.871940 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:36.872775 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:36.873009 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:36.873120 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:36.875840 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:50203: BlockingRecv error: recv got EOF from 127.12.12.2:50203 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 21 07:52:37 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:37 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.3@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:37.066524 21538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:37.060820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.3:60669)
0621 07:52:37.061118 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:37.061132 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:37.061231 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:37.061548 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:37.061551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:37.061738 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:37.061835 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:37.061839 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:37.062731 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:37.062735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:37.063348 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:37.063358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:37.063887 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:37.063890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:37.063959 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:37.064484 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:37.064496 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:37.066294 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":130,"threads_started":1}
W20260621 07:52:37.066630 21536 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:37.060964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:60669 (local address 127.12.12.62:38533)
0621 07:52:37.061100 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:37.061103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:37.061226 (+ 123us) server_negotiation.cc:407] Connection header received
0621 07:52:37.061301 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:37.061303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:37.061387 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:37.061463 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:37.061961 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:37.062614 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:37.063519 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:37.063764 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:37.063862 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:37.066517 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.3:60669: BlockingRecv error: recv got EOF from 127.12.12.3:60669 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 21 07:52:37 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:37 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.1@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:37.704720 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:37.698652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.1:38569)
0621 07:52:37.698938 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:37.698956 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:37.699066 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:37.699308 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:37.699310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:37.699518 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:37.699614 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:37.699619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:37.700487 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:37.700490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:37.701231 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:37.701255 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:37.701765 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:37.701769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:37.701846 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:37.702447 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:37.702458 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:37.704584 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":68,"threads_started":1}
W20260621 07:52:37.704947 21540 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:37.698700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:38569 (local address 127.12.12.62:38533)
0621 07:52:37.698972 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:37.698976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:37.699012 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:52:37.699153 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:37.699156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:37.699198 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:37.699277 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:37.699723 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:37.700380 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:37.701367 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:37.701662 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:37.701763 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:37.704780 (+ 3017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.1:38569: BlockingRecv error: recv got EOF from 127.12.12.1:38569 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":96,"threads_started":1}
Jun 21 07:52:37 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:52:37 dist-test-slave-lbpt krb5kdc[20482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.2@KRBTEST.COM for oryx/127.12.12.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:52:37.882953 21541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:37.876874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38533 (local address 127.12.12.2:56679)
0621 07:52:37.877184 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:37.877208 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:37.877402 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:37.877711 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:37.877714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:37.877938 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:37.878034 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:37.878040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:37.878902 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:37.878907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:37.879498 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:37.879506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:37.879988 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:37.879991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:37.880080 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:37.880605 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:37.880618 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:37.882779 (+ 2161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.62:38533: Server oryx/127.12.12.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":90,"threads_started":1}
W20260621 07:52:37.883342 21540 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:37.876966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:56679 (local address 127.12.12.62:38533)
0621 07:52:37.877141 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:37.877145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:37.877287 (+ 142us) server_negotiation.cc:407] Connection header received
0621 07:52:37.877535 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:37.877538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:37.877593 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:37.877684 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:37.878167 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:37.878779 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:37.879630 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:37.879867 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:37.879983 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:37.882990 (+ 3007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.2:56679: BlockingRecv error: recv got EOF from 127.12.12.2:56679 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 07:52:37.953649 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 21065
I20260621 07:52:37.959168 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 21217
I20260621 07:52:37.964455 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 21349
I20260621 07:52:37.969636 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 20995
2026-06-21T07:52:37Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18049 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-21T07:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:52:37Z Disabled control of system clock
I20260621 07:52:38.002151 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:38715
--webserver_interface=127.12.12.62
--webserver_port=0
--builtin_ntp_servers=127.12.12.20:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:38715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 07:52:38.113984 21545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:38.114272 21545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:38.114308 21545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:38.118556 21545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:38.118621 21545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:38.118637 21545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:38.118654 21545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:38.118703 21545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:52:38.123577 21545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:38715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:38715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:38.124723 21545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:38.126142 21545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:38.135152 21551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:38.135160 21550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:38.135162 21553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:38.135290 21545 server_base.cc:1034] running on GCE node
I20260621 07:52:38.136224 21545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:38.137344 21545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:38.138558 21545 hybrid_clock.cc:648] HybridClock initialized: now 1782028358138526 us; error 46 us; skew 500 ppm
I20260621 07:52:38.140614 21545 webserver.cc:460] Webserver started at http://127.12.12.62:43849/ using document root <none> and password file <none>
I20260621 07:52:38.141232 21545 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:38.141330 21545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:38.141564 21545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:38.143779 21545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "dfe97d7f222c4e72bceeaffc221d2bf4"
format_stamp: "Formatted at 2026-06-21 07:52:38 on dist-test-slave-lbpt"
server_key: "a118d1fe25956e5fa5918443e2888b20"
server_key_iv: "ad310b471f30b1282b7abb6655045816"
server_key_version: "encryptionkey@0"
I20260621 07:52:38.144385 21545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "dfe97d7f222c4e72bceeaffc221d2bf4"
format_stamp: "Formatted at 2026-06-21 07:52:38 on dist-test-slave-lbpt"
server_key: "a118d1fe25956e5fa5918443e2888b20"
server_key_iv: "ad310b471f30b1282b7abb6655045816"
server_key_version: "encryptionkey@0"
I20260621 07:52:38.148255 21545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:52:38.150775 21559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:38.152076 21545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 07:52:38.152215 21545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "dfe97d7f222c4e72bceeaffc221d2bf4"
format_stamp: "Formatted at 2026-06-21 07:52:38 on dist-test-slave-lbpt"
server_key: "a118d1fe25956e5fa5918443e2888b20"
server_key_iv: "ad310b471f30b1282b7abb6655045816"
server_key_version: "encryptionkey@0"
I20260621 07:52:38.152329 21545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:38.166533 21545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:38.167214 21545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:38.167411 21545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:38.174264 21545 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:38715
I20260621 07:52:38.174276 21611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:38715 every 8 connection(s)
I20260621 07:52:38.175244 21545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:52:38.178390 21612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:38.178699 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 21545
I20260621 07:52:38.178808 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:52:38.179119 12336 external_mini_cluster.cc:1428] Setting key 8b32fbd40fbf44758fbbae69c8a2a10a
I20260621 07:52:38.184942 21612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4: Bootstrap starting.
I20260621 07:52:38.187120 21612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:38.187956 21612 log.cc:824] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:38.188247 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:38.180479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.12.12.62:38715)
0621 07:52:38.181106 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:38.181122 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:38.181156 (+ 34us) server_negotiation.cc:407] Connection header received
0621 07:52:38.181812 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:38.181828 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:38.182210 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:38.182581 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:38.182993 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:38.184047 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:38.184951 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:38.185474 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:38.185645 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:38.186543 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:38.186548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:38.186560 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:38.186940 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:38.186996 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:38.187015 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:38.187121 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:38.187243 (+ 122us) server_negotiation.cc:299] Negotiation successful
0621 07:52:38.187498 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":130,"threads_started":1}
I20260621 07:52:38.189842 21612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4: No bootstrap required, opened a new log
I20260621 07:52:38.193318 21612 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe97d7f222c4e72bceeaffc221d2bf4" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38715 } }
I20260621 07:52:38.193576 21612 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:38.193655 21612 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe97d7f222c4e72bceeaffc221d2bf4, State: Initialized, Role: FOLLOWER
I20260621 07:52:38.194108 21612 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe97d7f222c4e72bceeaffc221d2bf4" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38715 } }
I20260621 07:52:38.194253 21612 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:52:38.194337 21612 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:52:38.194458 21612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:38.195348 21612 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe97d7f222c4e72bceeaffc221d2bf4" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38715 } }
I20260621 07:52:38.195696 21612 leader_election.cc:302] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfe97d7f222c4e72bceeaffc221d2bf4; no voters:
I20260621 07:52:38.196012 21612 leader_election.cc:288] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:52:38.196174 21617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:38.196489 21617 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [term 1 LEADER]: Becoming Leader. State: Replica: dfe97d7f222c4e72bceeaffc221d2bf4, State: Running, Role: LEADER
I20260621 07:52:38.196789 21617 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe97d7f222c4e72bceeaffc221d2bf4" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38715 } }
I20260621 07:52:38.197273 21612 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:52:38.198406 21619 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfe97d7f222c4e72bceeaffc221d2bf4. Latest consensus state: current_term: 1 leader_uuid: "dfe97d7f222c4e72bceeaffc221d2bf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe97d7f222c4e72bceeaffc221d2bf4" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38715 } } }
I20260621 07:52:38.198556 21619 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:38.198961 21626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:52:38.198920 21618 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfe97d7f222c4e72bceeaffc221d2bf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe97d7f222c4e72bceeaffc221d2bf4" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38715 } } }
I20260621 07:52:38.199159 21618 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4 [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:38.202221 21626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:52:38.208359 21626 catalog_manager.cc:1347] Generated new cluster ID: 019b064fa2644431a8b5430181a78c86
I20260621 07:52:38.208448 21626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:52:38.223026 21626 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:52:38.223989 21626 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:52:38.229092 21626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfe97d7f222c4e72bceeaffc221d2bf4: Generated new TSK 0
I20260621 07:52:38.229799 21626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:52:38.237416 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38715
--builtin_ntp_servers=127.12.12.20:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260621 07:52:38.360152 21636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:38.360451 21636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:38.360491 21636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:38.364732 21636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:38.364799 21636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:38.364856 21636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:52:38.369719 21636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:38.370903 21636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:38.372272 21636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:38.382017 21644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:38.382012 21641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:38.382017 21642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:38.382200 21636 server_base.cc:1034] running on GCE node
I20260621 07:52:38.382750 21636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:38.383450 21636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:38.384650 21636 hybrid_clock.cc:648] HybridClock initialized: now 1782028358384635 us; error 37 us; skew 500 ppm
I20260621 07:52:38.386665 21636 webserver.cc:460] Webserver started at http://127.12.12.1:33683/ using document root <none> and password file <none>
I20260621 07:52:38.387252 21636 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:38.387346 21636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:38.387559 21636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:38.389590 21636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "7908e555ab414c07b0bd0f7defe0ef36"
format_stamp: "Formatted at 2026-06-21 07:52:38 on dist-test-slave-lbpt"
server_key: "f42c112e3eaf2635873bc3860668372c"
server_key_iv: "cb39070a03485c7ee544240b0978639d"
server_key_version: "encryptionkey@0"
I20260621 07:52:38.390147 21636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "7908e555ab414c07b0bd0f7defe0ef36"
format_stamp: "Formatted at 2026-06-21 07:52:38 on dist-test-slave-lbpt"
server_key: "f42c112e3eaf2635873bc3860668372c"
server_key_iv: "cb39070a03485c7ee544240b0978639d"
server_key_version: "encryptionkey@0"
I20260621 07:52:38.393715 21636 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:52:38.396384 21650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:38.397765 21636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260621 07:52:38.397897 21636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "7908e555ab414c07b0bd0f7defe0ef36"
format_stamp: "Formatted at 2026-06-21 07:52:38 on dist-test-slave-lbpt"
server_key: "f42c112e3eaf2635873bc3860668372c"
server_key_iv: "cb39070a03485c7ee544240b0978639d"
server_key_version: "encryptionkey@0"
I20260621 07:52:38.398015 21636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:38.412139 21636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:38.412848 21636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:38.413060 21636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:38.413666 21636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:38.414706 21636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:38.414775 21636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:38.414848 21636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:38.414898 21636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:38.424077 21636 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:40091
I20260621 07:52:38.424099 21763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:40091 every 8 connection(s)
I20260621 07:52:38.425091 21636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:52:38.434186 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:38.426979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:53605 (local address 127.12.12.62:38715)
0621 07:52:38.427116 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:38.427120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:38.428196 (+ 1076us) server_negotiation.cc:407] Connection header received
0621 07:52:38.429630 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:38.429634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:38.429685 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:38.429768 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:38.430721 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:38.431289 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:38.432385 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:38.432611 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:38.432688 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:38.433752 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:38.433755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:38.433757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:38.433769 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:38.433788 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:38.433796 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:38.433863 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:38.434016 (+ 153us) server_negotiation.cc:299] Negotiation successful
0621 07:52:38.434069 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:52:38.434857 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 21636
I20260621 07:52:38.434978 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:52:38.435281 12336 external_mini_cluster.cc:1428] Setting key de063b0414850c1fad11e9ac2c421d06
I20260621 07:52:38.435323 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:38.427371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38715 (local address 127.12.12.1:53605)
0621 07:52:38.428041 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:38.428082 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:38.429363 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:38.429953 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:38.429959 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:38.430047 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:38.430583 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:38.430597 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:38.431392 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:38.431394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:38.432279 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:38.432288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:38.432746 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:38.432749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:38.432904 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:38.433481 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:52:38.433482 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:38.433522 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:52:38.433585 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:38.433875 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:38.433881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:38.433899 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 07:52:38.433994 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 07:52:38.434214 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
I20260621 07:52:38.436697 21764 heartbeater.cc:347] Connected to a master server at 127.12.12.62:38715
I20260621 07:52:38.436964 21764 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:38.437474 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38715
--builtin_ntp_servers=127.12.12.20:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 07:52:38.437542 21764 heartbeater.cc:511] Master 127.12.12.62:38715 requested a full tablet report, sending...
I20260621 07:52:38.439236 21575 ts_manager.cc:194] Registered new tserver with Master: 7908e555ab414c07b0bd0f7defe0ef36 (127.12.12.1:40091)
I20260621 07:52:38.440380 21575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.12.1:53605
W20260621 07:52:38.551509 21767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:38.551801 21767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:38.551841 21767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:38.556186 21767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:38.556286 21767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:38.556370 21767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:52:38.561370 21767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:38.562490 21767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:38.563827 21767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:38.573724 21772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:38.573891 21767 server_base.cc:1034] running on GCE node
W20260621 07:52:38.573732 21773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:38.573732 21775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:38.574602 21767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:38.575346 21767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:38.576577 21767 hybrid_clock.cc:648] HybridClock initialized: now 1782028358576557 us; error 38 us; skew 500 ppm
I20260621 07:52:38.578858 21767 webserver.cc:460] Webserver started at http://127.12.12.2:43415/ using document root <none> and password file <none>
I20260621 07:52:38.579507 21767 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:38.579613 21767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:38.579855 21767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:38.582024 21767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "a7f8bafe51aa41d0bd1fe788b1e5df49"
format_stamp: "Formatted at 2026-06-21 07:52:38 on dist-test-slave-lbpt"
server_key: "5f9b761ceddd377006fd318f58dd0728"
server_key_iv: "4078a26045ed3ff1c2f7d0cb257ac099"
server_key_version: "encryptionkey@0"
I20260621 07:52:38.582662 21767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "a7f8bafe51aa41d0bd1fe788b1e5df49"
format_stamp: "Formatted at 2026-06-21 07:52:38 on dist-test-slave-lbpt"
server_key: "5f9b761ceddd377006fd318f58dd0728"
server_key_iv: "4078a26045ed3ff1c2f7d0cb257ac099"
server_key_version: "encryptionkey@0"
I20260621 07:52:38.586840 21767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:52:38.589684 21781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:38.591432 21767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:38.591538 21767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "a7f8bafe51aa41d0bd1fe788b1e5df49"
format_stamp: "Formatted at 2026-06-21 07:52:38 on dist-test-slave-lbpt"
server_key: "5f9b761ceddd377006fd318f58dd0728"
server_key_iv: "4078a26045ed3ff1c2f7d0cb257ac099"
server_key_version: "encryptionkey@0"
I20260621 07:52:38.591639 21767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:38.614645 21767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:38.615332 21767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:38.615487 21767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:38.616036 21767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:38.616954 21767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:38.617002 21767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:38.617043 21767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:38.617067 21767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:38.626010 21767 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:33273
I20260621 07:52:38.626036 21894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:33273 every 8 connection(s)
I20260621 07:52:38.626967 21767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:52:38.634711 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 21767
I20260621 07:52:38.634873 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:52:38.635190 12336 external_mini_cluster.cc:1428] Setting key 75b15c36c7f71d5a2cd71ba572f72d02
I20260621 07:52:38.636773 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:38.628877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:55831 (local address 127.12.12.62:38715)
0621 07:52:38.629022 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:38.629026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:38.630190 (+ 1164us) server_negotiation.cc:407] Connection header received
0621 07:52:38.631703 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:38.631707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:38.631765 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:38.631878 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:38.632848 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:38.633377 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:38.634627 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:38.634890 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:38.635018 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:38.636252 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:38.636255 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:38.636257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:38.636269 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:38.636287 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:38.636295 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:38.636372 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:38.636570 (+ 198us) server_negotiation.cc:299] Negotiation successful
0621 07:52:38.636633 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:52:38.637425 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38715
--builtin_ntp_servers=127.12.12.20:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 07:52:38.637744 21897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:38.629301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38715 (local address 127.12.12.2:55831)
0621 07:52:38.630023 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:38.630065 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:38.631481 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:38.632064 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:38.632071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:38.632156 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:38.632693 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:38.632706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:38.633503 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:38.633513 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:38.634487 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:38.634497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:38.635073 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:38.635076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:38.635254 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:38.635983 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:52:38.635985 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:38.636026 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:52:38.636091 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:38.636428 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:38.636435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:38.636441 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:52:38.636570 (+ 129us) client_negotiation.cc:240] Negotiation successful
0621 07:52:38.636889 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":115,"threads_started":1}
I20260621 07:52:38.639288 21895 heartbeater.cc:347] Connected to a master server at 127.12.12.62:38715
I20260621 07:52:38.639571 21895 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:38.640272 21895 heartbeater.cc:511] Master 127.12.12.62:38715 requested a full tablet report, sending...
I20260621 07:52:38.641572 21575 ts_manager.cc:194] Registered new tserver with Master: a7f8bafe51aa41d0bd1fe788b1e5df49 (127.12.12.2:33273)
I20260621 07:52:38.642652 21575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.12.2:55831
W20260621 07:52:38.758061 21898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:38.758324 21898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:38.758360 21898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:38.762279 21898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:38.762356 21898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:38.762463 21898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:52:38.767326 21898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:38715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:38.768337 21898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:38.769831 21898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:38.779628 21904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:38.779632 21906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:38.779654 21903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:38.780042 21898 server_base.cc:1034] running on GCE node
I20260621 07:52:38.780488 21898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:38.781163 21898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:38.782395 21898 hybrid_clock.cc:648] HybridClock initialized: now 1782028358782362 us; error 49 us; skew 500 ppm
I20260621 07:52:38.784467 21898 webserver.cc:460] Webserver started at http://127.12.12.3:33413/ using document root <none> and password file <none>
I20260621 07:52:38.785085 21898 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:38.785183 21898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:38.785444 21898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:38.787691 21898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "4d6e19e15a484db5a02166902f114b7f"
format_stamp: "Formatted at 2026-06-21 07:52:38 on dist-test-slave-lbpt"
server_key: "f5f2a3789eb76f7ca8d9f6e99faafe60"
server_key_iv: "79faa1262226f503341a7a4b1fdb044d"
server_key_version: "encryptionkey@0"
I20260621 07:52:38.788260 21898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "4d6e19e15a484db5a02166902f114b7f"
format_stamp: "Formatted at 2026-06-21 07:52:38 on dist-test-slave-lbpt"
server_key: "f5f2a3789eb76f7ca8d9f6e99faafe60"
server_key_iv: "79faa1262226f503341a7a4b1fdb044d"
server_key_version: "encryptionkey@0"
I20260621 07:52:38.792274 21898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260621 07:52:38.795140 21912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:38.796506 21898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:52:38.796669 21898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "4d6e19e15a484db5a02166902f114b7f"
format_stamp: "Formatted at 2026-06-21 07:52:38 on dist-test-slave-lbpt"
server_key: "f5f2a3789eb76f7ca8d9f6e99faafe60"
server_key_iv: "79faa1262226f503341a7a4b1fdb044d"
server_key_version: "encryptionkey@0"
I20260621 07:52:38.796798 21898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:38.821403 21898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:38.822192 21898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:38.822461 21898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:38.823122 21898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:38.824119 21898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:38.824189 21898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:38.824255 21898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:38.824298 21898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:38.833664 21898 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:39079
I20260621 07:52:38.833714 22025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:39079 every 8 connection(s)
I20260621 07:52:38.834689 21898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:52:38.835325 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 21898
I20260621 07:52:38.835436 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:52:38.835719 12336 external_mini_cluster.cc:1428] Setting key dfd88952b49d455682f3dcc3b580d44a
I20260621 07:52:38.844089 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:38.836676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:39151 (local address 127.12.12.62:38715)
0621 07:52:38.836808 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:38.836813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:38.838135 (+ 1322us) server_negotiation.cc:407] Connection header received
0621 07:52:38.839504 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:38.839507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:38.839554 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:38.839629 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:38.840455 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:38.841008 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:38.842197 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:38.842528 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:38.842618 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:38.843672 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:38.843675 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:38.843677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:38.843689 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:38.843709 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:38.843717 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:38.843783 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:38.843920 (+ 137us) server_negotiation.cc:299] Negotiation successful
0621 07:52:38.843971 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260621 07:52:38.844970 22028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:38.837211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38715 (local address 127.12.12.3:39151)
0621 07:52:38.837968 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:38.838016 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:38.839305 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:38.839764 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:38.839771 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:38.839853 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:38.840329 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:38.840341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:38.841128 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:38.841131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:38.842068 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:38.842077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:38.842673 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:38.842675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:38.842815 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:38.843390 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:52:38.843392 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:38.843431 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:52:38.843526 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:38.843793 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:38.843799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:38.843803 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 07:52:38.843900 (+ 97us) client_negotiation.cc:240] Negotiation successful
0621 07:52:38.844126 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260621 07:52:38.846083 22026 heartbeater.cc:347] Connected to a master server at 127.12.12.62:38715
I20260621 07:52:38.846294 22026 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:38.846832 22026 heartbeater.cc:511] Master 127.12.12.62:38715 requested a full tablet report, sending...
I20260621 07:52:38.847882 21575 ts_manager.cc:194] Registered new tserver with Master: 4d6e19e15a484db5a02166902f114b7f (127.12.12.3:39079)
I20260621 07:52:38.848510 21575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.12.3:39151
I20260621 07:52:38.849995 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 07:52:38.853237 21615 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:38.852028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36284 (local address 127.12.12.62:38715)
0621 07:52:38.852203 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:38.852209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:38.852453 (+ 244us) server_negotiation.cc:407] Connection header received
0621 07:52:38.852622 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:38.852625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:38.852677 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:38.852758 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:38.853115 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36284: BlockingRecv error: recv got EOF from 127.0.0.1:36284 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 07:52:38.853438 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 21636
I20260621 07:52:38.859115 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 21767
I20260621 07:52:38.864308 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 21898
I20260621 07:52:38.869614 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 21545
2026-06-21T07:52:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (899 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-21T07:52:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:52:38Z Disabled control of system clock
I20260621 07:52:38.899257 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:35131
--webserver_interface=127.12.12.62
--webserver_port=0
--builtin_ntp_servers=127.12.12.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:35131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 07:52:39.012240 22037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:39.012533 22037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:39.012611 22037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:39.016948 22037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:39.017025 22037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:39.017041 22037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:39.017058 22037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:39.017073 22037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:52:39.021950 22037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:35131
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:35131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:39.023141 22037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:39.024586 22037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:39.033365 22042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:39.033443 22043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:39.033615 22037 server_base.cc:1034] running on GCE node
W20260621 07:52:39.033787 22045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:39.034461 22037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:39.035737 22037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:39.036916 22037 hybrid_clock.cc:648] HybridClock initialized: now 1782028359036887 us; error 42 us; skew 500 ppm
I20260621 07:52:39.039058 22037 webserver.cc:460] Webserver started at http://127.12.12.62:35791/ using document root <none> and password file <none>
I20260621 07:52:39.039680 22037 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:39.039744 22037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:39.039978 22037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:39.042193 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "020b1fcfd2654610ac874e1b1ef22f89"
format_stamp: "Formatted at 2026-06-21 07:52:39 on dist-test-slave-lbpt"
server_key: "38026d5e61c327d42387fa720f8ad513"
server_key_iv: "2cc465b95251f8f31d8fe4034088510e"
server_key_version: "encryptionkey@0"
I20260621 07:52:39.042843 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "020b1fcfd2654610ac874e1b1ef22f89"
format_stamp: "Formatted at 2026-06-21 07:52:39 on dist-test-slave-lbpt"
server_key: "38026d5e61c327d42387fa720f8ad513"
server_key_iv: "2cc465b95251f8f31d8fe4034088510e"
server_key_version: "encryptionkey@0"
I20260621 07:52:39.046723 22037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:52:39.049237 22051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:39.050730 22037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:39.050885 22037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "020b1fcfd2654610ac874e1b1ef22f89"
format_stamp: "Formatted at 2026-06-21 07:52:39 on dist-test-slave-lbpt"
server_key: "38026d5e61c327d42387fa720f8ad513"
server_key_iv: "2cc465b95251f8f31d8fe4034088510e"
server_key_version: "encryptionkey@0"
I20260621 07:52:39.051008 22037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:39.055589 22037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:39.056203 22037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:39.056394 22037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:39.063165 22037 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:35131
I20260621 07:52:39.063177 22103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:35131 every 8 connection(s)
I20260621 07:52:39.064109 22037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:52:39.065066 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 22037
I20260621 07:52:39.065182 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:52:39.065481 12336 external_mini_cluster.cc:1428] Setting key 122847744be90dfe09add05825a0ff39
I20260621 07:52:39.067670 22104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:39.071723 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:39.066949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47936 (local address 127.12.12.62:35131)
0621 07:52:39.068190 (+ 1241us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:39.068205 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:39.068241 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:52:39.068946 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:39.068964 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:39.069316 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:39.069660 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:39.070496 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:39.070501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:39.070512 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:39.070522 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:39.070578 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:39.070594 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:39.070743 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:39.070823 (+ 80us) server_negotiation.cc:299] Negotiation successful
0621 07:52:39.071055 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":663,"threads_started":1}
I20260621 07:52:39.073679 22104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89: Bootstrap starting.
I20260621 07:52:39.076007 22104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:39.076920 22104 log.cc:824] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:39.079020 22104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89: No bootstrap required, opened a new log
I20260621 07:52:39.082724 22104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020b1fcfd2654610ac874e1b1ef22f89" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 35131 } }
I20260621 07:52:39.082954 22104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:39.082998 22104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 020b1fcfd2654610ac874e1b1ef22f89, State: Initialized, Role: FOLLOWER
I20260621 07:52:39.083365 22104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020b1fcfd2654610ac874e1b1ef22f89" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 35131 } }
I20260621 07:52:39.083480 22104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:52:39.083518 22104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:52:39.083587 22104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:39.084447 22104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020b1fcfd2654610ac874e1b1ef22f89" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 35131 } }
I20260621 07:52:39.084734 22104 leader_election.cc:302] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 020b1fcfd2654610ac874e1b1ef22f89; no voters:
I20260621 07:52:39.084983 22104 leader_election.cc:288] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:52:39.085137 22109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:39.085376 22109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [term 1 LEADER]: Becoming Leader. State: Replica: 020b1fcfd2654610ac874e1b1ef22f89, State: Running, Role: LEADER
I20260621 07:52:39.085676 22109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020b1fcfd2654610ac874e1b1ef22f89" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 35131 } }
I20260621 07:52:39.086341 22104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:52:39.087569 22110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "020b1fcfd2654610ac874e1b1ef22f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020b1fcfd2654610ac874e1b1ef22f89" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 35131 } } }
I20260621 07:52:39.087711 22110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:39.087569 22111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 020b1fcfd2654610ac874e1b1ef22f89. Latest consensus state: current_term: 1 leader_uuid: "020b1fcfd2654610ac874e1b1ef22f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020b1fcfd2654610ac874e1b1ef22f89" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 35131 } } }
I20260621 07:52:39.087882 22111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89 [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:39.088137 22118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:52:39.091368 22118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:52:39.097039 22118 catalog_manager.cc:1347] Generated new cluster ID: 7725a8fbbed84988805deb01ac24ab18
I20260621 07:52:39.097118 22118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:52:39.127015 22118 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:52:39.127377 22118 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:52:39.145054 22118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 020b1fcfd2654610ac874e1b1ef22f89: Generated new TSK 0
I20260621 07:52:39.146198 22118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:52:39.153694 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:35131
--builtin_ntp_servers=127.12.12.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 07:52:39.284034 22128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:39.284335 22128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:39.284413 22128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:39.288950 22128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:39.289028 22128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:39.289117 22128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:52:39.294083 22128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:35131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:39.295222 22128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:39.296674 22128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:39.306226 22133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:39.306239 22134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:39.306239 22136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:39.306346 22128 server_base.cc:1034] running on GCE node
I20260621 07:52:39.306994 22128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:39.307679 22128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:39.308862 22128 hybrid_clock.cc:648] HybridClock initialized: now 1782028359308828 us; error 48 us; skew 500 ppm
I20260621 07:52:39.310823 22128 webserver.cc:460] Webserver started at http://127.12.12.1:45323/ using document root <none> and password file <none>
I20260621 07:52:39.311398 22128 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:39.311460 22128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:39.311712 22128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:39.313741 22128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "10f37477b0224196b5cd565b81cf4eb0"
format_stamp: "Formatted at 2026-06-21 07:52:39 on dist-test-slave-lbpt"
server_key: "df41e7ccbb175d3f58bf5d41e569ed03"
server_key_iv: "e27371893637ce0d27a290239d14afce"
server_key_version: "encryptionkey@0"
I20260621 07:52:39.314291 22128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "10f37477b0224196b5cd565b81cf4eb0"
format_stamp: "Formatted at 2026-06-21 07:52:39 on dist-test-slave-lbpt"
server_key: "df41e7ccbb175d3f58bf5d41e569ed03"
server_key_iv: "e27371893637ce0d27a290239d14afce"
server_key_version: "encryptionkey@0"
I20260621 07:52:39.317905 22128 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:52:39.320578 22142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:39.322161 22128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 07:52:39.322289 22128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "10f37477b0224196b5cd565b81cf4eb0"
format_stamp: "Formatted at 2026-06-21 07:52:39 on dist-test-slave-lbpt"
server_key: "df41e7ccbb175d3f58bf5d41e569ed03"
server_key_iv: "e27371893637ce0d27a290239d14afce"
server_key_version: "encryptionkey@0"
I20260621 07:52:39.322458 22128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:39.327068 22128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:39.327673 22128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:39.327857 22128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:39.328397 22128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:39.329296 22128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:39.329366 22128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:39.329427 22128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:39.329468 22128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:39.338047 22128 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:41397
I20260621 07:52:39.338068 22255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:41397 every 8 connection(s)
I20260621 07:52:39.339043 22128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:52:39.341012 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 22128
I20260621 07:52:39.341133 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:52:39.341383 12336 external_mini_cluster.cc:1428] Setting key f56bcde6913d77157295776bcf43c729
I20260621 07:52:39.343708 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:35131
--builtin_ntp_servers=127.12.12.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260621 07:52:39.346119 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:39.341000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:45001 (local address 127.12.12.62:35131)
0621 07:52:39.341160 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:39.341165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:39.342345 (+ 1180us) server_negotiation.cc:407] Connection header received
0621 07:52:39.344188 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:39.344191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:39.344225 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:39.344311 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:39.345563 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:39.345569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:39.345573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:39.345590 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:39.345615 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:39.345628 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:39.345726 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:39.345878 (+ 152us) server_negotiation.cc:299] Negotiation successful
0621 07:52:39.345925 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:52:39.347190 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:39.341399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:35131 (local address 127.12.12.1:45001)
0621 07:52:39.342204 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:39.342246 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:39.343948 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:39.344527 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:39.344539 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:39.344658 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:39.345258 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:52:39.345262 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:39.345321 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:52:39.345397 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:39.345726 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:39.345734 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:39.345738 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 07:52:39.345884 (+ 146us) client_negotiation.cc:240] Negotiation successful
0621 07:52:39.346116 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":101,"threads_started":1}
I20260621 07:52:39.348470 22256 heartbeater.cc:347] Connected to a master server at 127.12.12.62:35131
I20260621 07:52:39.348737 22256 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:39.349257 22256 heartbeater.cc:511] Master 127.12.12.62:35131 requested a full tablet report, sending...
I20260621 07:52:39.350968 22068 ts_manager.cc:194] Registered new tserver with Master: 10f37477b0224196b5cd565b81cf4eb0 (127.12.12.1:41397)
W20260621 07:52:39.458236 22259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:39.458582 22259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:39.458667 22259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:39.462817 22259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:39.462908 22259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:39.463008 22259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:52:39.467867 22259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:35131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:39.469027 22259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:39.470556 22259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:39.480067 22267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:39.480165 22259 server_base.cc:1034] running on GCE node
W20260621 07:52:39.480057 22265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:39.480175 22264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:39.480818 22259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:39.481483 22259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:39.482671 22259 hybrid_clock.cc:648] HybridClock initialized: now 1782028359482632 us; error 52 us; skew 500 ppm
I20260621 07:52:39.484584 22259 webserver.cc:460] Webserver started at http://127.12.12.2:36329/ using document root <none> and password file <none>
I20260621 07:52:39.485167 22259 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:39.485260 22259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:39.485569 22259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:39.488057 22259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "70cc553a87aa46edb3e1a7796558520d"
format_stamp: "Formatted at 2026-06-21 07:52:39 on dist-test-slave-lbpt"
server_key: "e7f44096d55518a2dea9aa6f43297ba2"
server_key_iv: "9685989622096f430736e34d26632083"
server_key_version: "encryptionkey@0"
I20260621 07:52:39.488677 22259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "70cc553a87aa46edb3e1a7796558520d"
format_stamp: "Formatted at 2026-06-21 07:52:39 on dist-test-slave-lbpt"
server_key: "e7f44096d55518a2dea9aa6f43297ba2"
server_key_iv: "9685989622096f430736e34d26632083"
server_key_version: "encryptionkey@0"
I20260621 07:52:39.492573 22259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:52:39.495115 22273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:39.496371 22259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:52:39.496491 22259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "70cc553a87aa46edb3e1a7796558520d"
format_stamp: "Formatted at 2026-06-21 07:52:39 on dist-test-slave-lbpt"
server_key: "e7f44096d55518a2dea9aa6f43297ba2"
server_key_iv: "9685989622096f430736e34d26632083"
server_key_version: "encryptionkey@0"
I20260621 07:52:39.496623 22259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:39.501147 22259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:39.501734 22259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:39.501915 22259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:39.502516 22259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:39.503415 22259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:39.503492 22259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:39.503559 22259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:39.503592 22259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:39.512369 22259 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:46119
I20260621 07:52:39.512391 22386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:46119 every 8 connection(s)
I20260621 07:52:39.513311 22259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:52:39.519578 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:39.515314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:53487 (local address 127.12.12.62:35131)
0621 07:52:39.515507 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:39.515512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:39.516681 (+ 1169us) server_negotiation.cc:407] Connection header received
0621 07:52:39.517958 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:39.517960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:39.517997 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:39.518074 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:39.519154 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:39.519157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:39.519159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:39.519171 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:39.519190 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:39.519199 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:39.519279 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:39.519431 (+ 152us) server_negotiation.cc:299] Negotiation successful
0621 07:52:39.519468 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260621 07:52:39.520471 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:39.515791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:35131 (local address 127.12.12.2:53487)
0621 07:52:39.516529 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:39.516570 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:39.517760 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:39.518207 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:39.518214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:39.518310 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:39.518900 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:52:39.518901 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:39.518942 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:52:39.519014 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:39.519275 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:39.519281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:39.519285 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 07:52:39.519455 (+ 170us) client_negotiation.cc:240] Negotiation successful
0621 07:52:39.519687 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":114,"threads_started":1}
I20260621 07:52:39.521054 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 22259
I20260621 07:52:39.521183 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:52:39.521481 12336 external_mini_cluster.cc:1428] Setting key cdde6abcff7f3288f483804569035188
I20260621 07:52:39.521767 22387 heartbeater.cc:347] Connected to a master server at 127.12.12.62:35131
I20260621 07:52:39.521999 22387 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:39.522511 22387 heartbeater.cc:511] Master 127.12.12.62:35131 requested a full tablet report, sending...
I20260621 07:52:39.523499 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:35131
--builtin_ntp_servers=127.12.12.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260621 07:52:39.523595 22068 ts_manager.cc:194] Registered new tserver with Master: 70cc553a87aa46edb3e1a7796558520d (127.12.12.2:46119)
W20260621 07:52:39.637292 22390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:39.637583 22390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:39.637625 22390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:39.642381 22390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:39.642500 22390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:39.642570 22390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:52:39.647769 22390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:35131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:39.648845 22390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:39.650274 22390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:39.660354 22398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:39.660354 22395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:39.660482 22390 server_base.cc:1034] running on GCE node
W20260621 07:52:39.660364 22396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:39.661157 22390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:39.661880 22390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:39.663061 22390 hybrid_clock.cc:648] HybridClock initialized: now 1782028359663044 us; error 31 us; skew 500 ppm
I20260621 07:52:39.665001 22390 webserver.cc:460] Webserver started at http://127.12.12.3:36115/ using document root <none> and password file <none>
I20260621 07:52:39.665609 22390 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:39.665701 22390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:39.665934 22390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:39.668100 22390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "ea57e30c18d24e5b81b55919f77acb6c"
format_stamp: "Formatted at 2026-06-21 07:52:39 on dist-test-slave-lbpt"
server_key: "5801c50b1bb4c98e0bd1d21735774250"
server_key_iv: "badf3b99de2a7a41a8a06745b6ef63d2"
server_key_version: "encryptionkey@0"
I20260621 07:52:39.668687 22390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "ea57e30c18d24e5b81b55919f77acb6c"
format_stamp: "Formatted at 2026-06-21 07:52:39 on dist-test-slave-lbpt"
server_key: "5801c50b1bb4c98e0bd1d21735774250"
server_key_iv: "badf3b99de2a7a41a8a06745b6ef63d2"
server_key_version: "encryptionkey@0"
I20260621 07:52:39.672662 22390 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:52:39.675369 22404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:39.676586 22390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260621 07:52:39.676723 22390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "ea57e30c18d24e5b81b55919f77acb6c"
format_stamp: "Formatted at 2026-06-21 07:52:39 on dist-test-slave-lbpt"
server_key: "5801c50b1bb4c98e0bd1d21735774250"
server_key_iv: "badf3b99de2a7a41a8a06745b6ef63d2"
server_key_version: "encryptionkey@0"
I20260621 07:52:39.676847 22390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:39.681723 22390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:39.682363 22390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:39.682595 22390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:39.683143 22390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:39.684078 22390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:39.684145 22390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:39.684214 22390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:39.684254 22390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:39.693919 22390 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:45273
I20260621 07:52:39.693964 22517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:45273 every 8 connection(s)
I20260621 07:52:39.694972 22390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:52:39.699983 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 22390
I20260621 07:52:39.700083 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:52:39.700413 12336 external_mini_cluster.cc:1428] Setting key 722bef21319ee3a421fbf83d1f5d687a
I20260621 07:52:39.701814 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:39.696927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:53659 (local address 127.12.12.62:35131)
0621 07:52:39.697085 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:39.697089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:39.698203 (+ 1114us) server_negotiation.cc:407] Connection header received
0621 07:52:39.699709 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:39.699713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:39.699764 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:39.699873 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:39.701194 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:39.701198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:39.701200 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:52:39.701219 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:39.701239 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:52:39.701248 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:39.701367 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:39.701604 (+ 237us) server_negotiation.cc:299] Negotiation successful
0621 07:52:39.701656 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 07:52:39.702831 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:39.697384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:35131 (local address 127.12.12.3:53659)
0621 07:52:39.698056 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:39.698097 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:39.699401 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:39.700029 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:39.700064 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:39.700206 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:39.700892 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:52:39.700894 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:39.700953 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:52:39.701040 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:39.701373 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:39.701384 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:39.701390 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:52:39.701602 (+ 212us) client_negotiation.cc:240] Negotiation successful
0621 07:52:39.701851 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":108,"threads_started":1}
I20260621 07:52:39.704078 22518 heartbeater.cc:347] Connected to a master server at 127.12.12.62:35131
I20260621 07:52:39.704356 22518 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:39.704792 22518 heartbeater.cc:511] Master 127.12.12.62:35131 requested a full tablet report, sending...
I20260621 07:52:39.705848 22068 ts_manager.cc:194] Registered new tserver with Master: ea57e30c18d24e5b81b55919f77acb6c (127.12.12.3:45273)
I20260621 07:52:39.714865 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 07:52:39.717881 22107 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:52:39.716896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47950 (local address 127.12.12.62:35131)
0621 07:52:39.717061 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:39.717066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:39.717216 (+ 150us) server_negotiation.cc:407] Connection header received
0621 07:52:39.717406 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:39.717409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:39.717444 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:39.717536 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:39.717777 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47950: BlockingRecv error: recv got EOF from 127.0.0.1:47950 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:52:39.718372 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 22128
I20260621 07:52:39.723726 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 22259
I20260621 07:52:39.729100 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 22390
I20260621 07:52:39.734195 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 22037
2026-06-21T07:52:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (863 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:52:39 dist-test-slave-lbpt krb5kdc[22528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:52:39 dist-test-slave-lbpt krb5kdc[22528](info): set up 2 sockets
Jun 21 07:52:39 dist-test-slave-lbpt krb5kdc[22528](info): commencing operation
krb5kdc: starting...
W20260621 07:52:40.779459 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:52:40 dist-test-slave-lbpt krb5kdc[22528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:52:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:52:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:52:40.901875 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:39261
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:39261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:41.013026 22541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:41.013330 22541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:41.013372 22541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:41.017661 22541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:52:41.017724 22541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:41.017745 22541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:41.017762 22541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:52:41.017778 22541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:52:41.023247 22541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:39261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:39261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:41.024274 22541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:41.025265 22541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:41.034041 22547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:41.034055 22549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:41.034050 22546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:41.034399 22541 server_base.cc:1034] running on GCE node
I20260621 07:52:41.035027 22541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:41.036186 22541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:41.037380 22541 hybrid_clock.cc:648] HybridClock initialized: now 1782028361037365 us; error 31 us; skew 500 ppm
Jun 21 07:52:41 dist-test-slave-lbpt krb5kdc[22528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028361, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:41.040189 22541 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:52:41.041141 22541 webserver.cc:460] Webserver started at http://127.12.12.62:37637/ using document root <none> and password file <none>
I20260621 07:52:41.041728 22541 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:41.041821 22541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:41.042068 22541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:41.044306 22541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "f8a5c4549f494b39af8b7df64be95704"
format_stamp: "Formatted at 2026-06-21 07:52:41 on dist-test-slave-lbpt"
server_key: "e784b265fd1619157b234f9d74c02b21"
server_key_iv: "f688d3d8bb8442845a2c9620870792c4"
server_key_version: "encryptionkey@0"
I20260621 07:52:41.044920 22541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "f8a5c4549f494b39af8b7df64be95704"
format_stamp: "Formatted at 2026-06-21 07:52:41 on dist-test-slave-lbpt"
server_key: "e784b265fd1619157b234f9d74c02b21"
server_key_iv: "f688d3d8bb8442845a2c9620870792c4"
server_key_version: "encryptionkey@0"
I20260621 07:52:41.049918 22541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:52:41.052886 22556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:41.054286 22541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:52:41.054525 22541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "f8a5c4549f494b39af8b7df64be95704"
format_stamp: "Formatted at 2026-06-21 07:52:41 on dist-test-slave-lbpt"
server_key: "e784b265fd1619157b234f9d74c02b21"
server_key_iv: "f688d3d8bb8442845a2c9620870792c4"
server_key_version: "encryptionkey@0"
I20260621 07:52:41.054662 22541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:41.088477 22541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:41.092409 22541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:41.092595 22541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:41.099751 22541 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:39261
I20260621 07:52:41.100068 22608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:39261 every 8 connection(s)
I20260621 07:52:41.100790 22541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:52:41.103817 22609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:41.109015 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 22541
I20260621 07:52:41.109118 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:52:41.109421 12336 external_mini_cluster.cc:1428] Setting key cdae984fd73c333f510965b75eea010b
I20260621 07:52:41.110747 22609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704: Bootstrap starting.
I20260621 07:52:41.113162 22609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:41.113960 22609 log.cc:824] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:41.116005 22609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704: No bootstrap required, opened a new log
Jun 21 07:52:41 dist-test-slave-lbpt krb5kdc[22528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:41.119980 22609 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a5c4549f494b39af8b7df64be95704" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39261 } }
I20260621 07:52:41.120334 22609 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:41.120424 22609 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8a5c4549f494b39af8b7df64be95704, State: Initialized, Role: FOLLOWER
I20260621 07:52:41.120985 22609 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a5c4549f494b39af8b7df64be95704" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39261 } }
I20260621 07:52:41.121161 22609 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:52:41.121253 22609 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:52:41.121374 22609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:41.122329 22609 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a5c4549f494b39af8b7df64be95704" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39261 } }
I20260621 07:52:41.122735 22609 leader_election.cc:302] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8a5c4549f494b39af8b7df64be95704; no voters:
I20260621 07:52:41.123107 22609 leader_election.cc:288] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:52:41.123179 22614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:41.123386 22614 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [term 1 LEADER]: Becoming Leader. State: Replica: f8a5c4549f494b39af8b7df64be95704, State: Running, Role: LEADER
I20260621 07:52:41.123754 22614 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a5c4549f494b39af8b7df64be95704" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39261 } }
I20260621 07:52:41.124473 22609 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:52:41.125075 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:41.111027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55900 (local address 127.12.12.62:39261)
0621 07:52:41.111535 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:41.111552 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:41.111592 (+ 40us) server_negotiation.cc:407] Connection header received
0621 07:52:41.112291 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:41.112310 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:41.112667 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:41.113051 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:41.113872 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.115003 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.115894 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.116448 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.116622 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:41.119564 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:41.119570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:41.119587 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:41.119618 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:41.121733 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:41.122245 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:41.122251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:41.122257 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:41.122389 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:41.122736 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:41.122741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:41.122743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:41.123261 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:41.123503 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:41.123795 (+ 292us) server_negotiation.cc:299] Negotiation successful
0621 07:52:41.124158 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":136,"threads_started":1}
I20260621 07:52:41.125600 22615 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8a5c4549f494b39af8b7df64be95704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a5c4549f494b39af8b7df64be95704" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39261 } } }
I20260621 07:52:41.126475 22615 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [sys.catalog]: This master's current role is: LEADER
I20260621 07:52:41.126739 22615 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8a5c4549f494b39af8b7df64be95704. Latest consensus state: current_term: 1 leader_uuid: "f8a5c4549f494b39af8b7df64be95704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a5c4549f494b39af8b7df64be95704" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 39261 } } }
I20260621 07:52:41.127079 22615 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704 [sys.catalog]: This master's current role is: LEADER
W20260621 07:52:41.130168 22630 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:52:41.130252 22630 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:52:41.130383 22628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:52:41.131414 22628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:52:41.137260 22628 catalog_manager.cc:1347] Generated new cluster ID: 464cf2c7985f4372802086879109550a
I20260621 07:52:41.137346 22628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:52:41.195974 22628 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:52:41.197300 22628 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:52:41.207831 22628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704: Generated new TSK 0
I20260621 07:52:41.208760 22628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:52:41.241797 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:39261
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:41.355904 22636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:41.356213 22636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:41.356261 22636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:41.360728 22636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:41.360800 22636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:41.360869 22636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:52:41.366221 22636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:39261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:41.367401 22636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:41.368397 22636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:41.378351 22642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:41.378341 22641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:41.378332 22644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:41.378482 22636 server_base.cc:1034] running on GCE node
I20260621 07:52:41.379009 22636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:41.379739 22636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:41.380975 22636 hybrid_clock.cc:648] HybridClock initialized: now 1782028361380938 us; error 49 us; skew 500 ppm
Jun 21 07:52:41 dist-test-slave-lbpt krb5kdc[22528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028361, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:41.384083 22636 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:52:41.385042 22636 webserver.cc:460] Webserver started at http://127.12.12.1:38849/ using document root <none> and password file <none>
I20260621 07:52:41.385640 22636 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:41.385738 22636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:41.385980 22636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:41.388170 22636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "98a503e2b4cf47a99e31681df1868956"
format_stamp: "Formatted at 2026-06-21 07:52:41 on dist-test-slave-lbpt"
server_key: "49708a5242c44512ed8933512fc92d9d"
server_key_iv: "eb61d00a783b84277191c3114cc8769c"
server_key_version: "encryptionkey@0"
I20260621 07:52:41.388763 22636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "98a503e2b4cf47a99e31681df1868956"
format_stamp: "Formatted at 2026-06-21 07:52:41 on dist-test-slave-lbpt"
server_key: "49708a5242c44512ed8933512fc92d9d"
server_key_iv: "eb61d00a783b84277191c3114cc8769c"
server_key_version: "encryptionkey@0"
I20260621 07:52:41.392866 22636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260621 07:52:41.395462 22651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:41.396776 22636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:52:41.396938 22636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "98a503e2b4cf47a99e31681df1868956"
format_stamp: "Formatted at 2026-06-21 07:52:41 on dist-test-slave-lbpt"
server_key: "49708a5242c44512ed8933512fc92d9d"
server_key_iv: "eb61d00a783b84277191c3114cc8769c"
server_key_version: "encryptionkey@0"
I20260621 07:52:41.397061 22636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:41.407294 22636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:41.410737 22636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:41.410944 22636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:41.411538 22636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:41.412350 22636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:41.412421 22636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:41.412495 22636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:41.412539 22636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:41.421454 22636 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:39621
I20260621 07:52:41.421473 22764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:39621 every 8 connection(s)
I20260621 07:52:41.422374 22636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:52:41.428609 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 22636
I20260621 07:52:41.428709 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:52:41.429009 12336 external_mini_cluster.cc:1428] Setting key 635aa07868ee6f38c7a3197b05e307b7
Jun 21 07:52:41 dist-test-slave-lbpt krb5kdc[22528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028361, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:41.437772 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:41.424483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:33975 (local address 127.12.12.62:39261)
0621 07:52:41.424625 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:41.424629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:41.425611 (+ 982us) server_negotiation.cc:407] Connection header received
0621 07:52:41.427085 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:41.427088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:41.427135 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:41.427215 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:41.428384 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.429005 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.430478 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.430821 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.430922 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:41.433775 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:41.433778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:41.433780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:41.433801 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:41.435484 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:41.436145 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:41.436147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:41.436149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:41.436226 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:41.436630 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:41.436632 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:41.436634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:41.436804 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:41.436887 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:41.437538 (+ 651us) server_negotiation.cc:299] Negotiation successful
0621 07:52:41.437645 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
I20260621 07:52:41.439042 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:41.424914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39261 (local address 127.12.12.1:33975)
0621 07:52:41.425456 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:41.425500 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:41.426878 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:41.427381 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:41.427388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:41.427775 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:41.428243 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:41.428255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.429152 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.429156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:41.430320 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:41.430334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.430991 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.430997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:41.431231 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:41.431887 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:41.431913 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:41.433588 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:41.435609 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:41.435617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:41.435636 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:41.436034 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:41.436323 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:41.436330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:41.436333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:41.436529 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:41.436918 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:41.436925 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:41.437266 (+ 341us) client_negotiation.cc:769] Sending connection context
0621 07:52:41.437507 (+ 241us) client_negotiation.cc:240] Negotiation successful
0621 07:52:41.437865 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":126,"threads_started":1}
I20260621 07:52:41.440413 22765 heartbeater.cc:347] Connected to a master server at 127.12.12.62:39261
I20260621 07:52:41.440639 22765 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:41.441138 22765 heartbeater.cc:511] Master 127.12.12.62:39261 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
I20260621 07:52:41.442940 22573 ts_manager.cc:194] Registered new tserver with Master: 98a503e2b4cf47a99e31681df1868956 (127.12.12.1:39621)
I20260621 07:52:41.444235 22573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:33975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:52:41.455560 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:39261
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:41.569692 22770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:41.570012 22770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:41.570089 22770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:41.574545 22770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:41.574636 22770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:41.574745 22770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:52:41.580303 22770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:39261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:41.581576 22770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:41.582717 22770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:41.592473 22776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:41.592491 22778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:41.592532 22775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:41.592935 22770 server_base.cc:1034] running on GCE node
I20260621 07:52:41.593406 22770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:41.594053 22770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:41.595258 22770 hybrid_clock.cc:648] HybridClock initialized: now 1782028361595210 us; error 60 us; skew 500 ppm
Jun 21 07:52:41 dist-test-slave-lbpt krb5kdc[22528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028361, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:41.598142 22770 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:52:41.599084 22770 webserver.cc:460] Webserver started at http://127.12.12.2:45091/ using document root <none> and password file <none>
I20260621 07:52:41.599625 22770 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:41.599686 22770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:41.599851 22770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:41.602216 22770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "7691bcd947414b0bb234e7f7dd92b5c3"
format_stamp: "Formatted at 2026-06-21 07:52:41 on dist-test-slave-lbpt"
server_key: "46893f595cf74e900e5b14a26164bc7b"
server_key_iv: "5b15bc7e6b2eeca552bb2fd9b721919f"
server_key_version: "encryptionkey@0"
I20260621 07:52:41.602816 22770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "7691bcd947414b0bb234e7f7dd92b5c3"
format_stamp: "Formatted at 2026-06-21 07:52:41 on dist-test-slave-lbpt"
server_key: "46893f595cf74e900e5b14a26164bc7b"
server_key_iv: "5b15bc7e6b2eeca552bb2fd9b721919f"
server_key_version: "encryptionkey@0"
I20260621 07:52:41.606678 22770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:52:41.609203 22785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:41.610605 22770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:52:41.610737 22770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "7691bcd947414b0bb234e7f7dd92b5c3"
format_stamp: "Formatted at 2026-06-21 07:52:41 on dist-test-slave-lbpt"
server_key: "46893f595cf74e900e5b14a26164bc7b"
server_key_iv: "5b15bc7e6b2eeca552bb2fd9b721919f"
server_key_version: "encryptionkey@0"
I20260621 07:52:41.610847 22770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:41.628439 22770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:41.631857 22770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:41.632081 22770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:41.632716 22770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:41.633635 22770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:41.633683 22770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:41.633751 22770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:41.633782 22770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:41.642786 22770 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:41153
I20260621 07:52:41.642819 22898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:41153 every 8 connection(s)
I20260621 07:52:41.643707 22770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:52:41.652200 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 22770
I20260621 07:52:41.652323 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:52:41.652693 12336 external_mini_cluster.cc:1428] Setting key 6ca3157376dd64ba24713e884b4e9651
Jun 21 07:52:41 dist-test-slave-lbpt krb5kdc[22528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028361, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:41.658778 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:41.645544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:59321 (local address 127.12.12.62:39261)
0621 07:52:41.645690 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:41.645694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:41.646612 (+ 918us) server_negotiation.cc:407] Connection header received
0621 07:52:41.648093 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:41.648096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:41.648155 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:41.648271 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:41.649499 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.650042 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.651322 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.651559 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.651676 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:41.654732 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:41.654739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:41.654745 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:41.654778 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:41.656454 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:41.657103 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:41.657105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:41.657106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:41.657180 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:41.657569 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:41.657571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:41.657572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:41.657735 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:41.657807 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:41.658524 (+ 717us) server_negotiation.cc:299] Negotiation successful
0621 07:52:41.658629 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:52:41.659981 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:41.645951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39261 (local address 127.12.12.2:59321)
0621 07:52:41.646405 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:41.646505 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:41.647844 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:41.648456 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:41.648463 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:41.648844 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:41.649335 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:41.649349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.650173 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.650182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:41.651188 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:41.651198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.651720 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.651724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:41.651921 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:41.652640 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:41.652662 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:41.654570 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:41.656571 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:41.656580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:41.656600 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:41.656976 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:41.657269 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:41.657274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:41.657275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:41.657470 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:41.657839 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:41.657849 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:41.658194 (+ 345us) client_negotiation.cc:769] Sending connection context
0621 07:52:41.658406 (+ 212us) client_negotiation.cc:240] Negotiation successful
0621 07:52:41.658822 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
I20260621 07:52:41.661250 22899 heartbeater.cc:347] Connected to a master server at 127.12.12.62:39261
I20260621 07:52:41.661509 22899 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:41.662025 22899 heartbeater.cc:511] Master 127.12.12.62:39261 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
I20260621 07:52:41.663133 22573 ts_manager.cc:194] Registered new tserver with Master: 7691bcd947414b0bb234e7f7dd92b5c3 (127.12.12.2:41153)
I20260621 07:52:41.663722 22573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:59321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:52:41.679390 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:39261
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:52:41.794389 22904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:52:41.794668 22904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:52:41.794739 22904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:52:41.798810 22904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:52:41.798864 22904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:52:41.798956 22904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:52:41.804286 22904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:39261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:52:41.805344 22904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:52:41.806366 22904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:52:41.816772 22912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:41.816845 22904 server_base.cc:1034] running on GCE node
W20260621 07:52:41.816772 22909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:52:41.816792 22910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:52:41.817368 22904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:52:41.818125 22904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:52:41.819335 22904 hybrid_clock.cc:648] HybridClock initialized: now 1782028361819283 us; error 59 us; skew 500 ppm
Jun 21 07:52:41 dist-test-slave-lbpt krb5kdc[22528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028361, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:52:41.825706 22904 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:52:41.826874 22904 webserver.cc:460] Webserver started at http://127.12.12.3:36265/ using document root <none> and password file <none>
I20260621 07:52:41.827493 22904 fs_manager.cc:359] Metadata directory not provided
I20260621 07:52:41.827589 22904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:52:41.827831 22904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:52:41.829890 22904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "c86771082c084c1f89eb54d1c4dcc2cd"
format_stamp: "Formatted at 2026-06-21 07:52:41 on dist-test-slave-lbpt"
server_key: "55f93c1a985925ccc828b5a07d868198"
server_key_iv: "79b5f921a4d71e6140c2dd36dd2a4951"
server_key_version: "encryptionkey@0"
I20260621 07:52:41.830535 22904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "c86771082c084c1f89eb54d1c4dcc2cd"
format_stamp: "Formatted at 2026-06-21 07:52:41 on dist-test-slave-lbpt"
server_key: "55f93c1a985925ccc828b5a07d868198"
server_key_iv: "79b5f921a4d71e6140c2dd36dd2a4951"
server_key_version: "encryptionkey@0"
I20260621 07:52:41.834380 22904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:52:41.836911 22919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:41.838218 22904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:52:41.838382 22904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "c86771082c084c1f89eb54d1c4dcc2cd"
format_stamp: "Formatted at 2026-06-21 07:52:41 on dist-test-slave-lbpt"
server_key: "55f93c1a985925ccc828b5a07d868198"
server_key_iv: "79b5f921a4d71e6140c2dd36dd2a4951"
server_key_version: "encryptionkey@0"
I20260621 07:52:41.838553 22904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:52:41.852793 22904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:52:41.856112 22904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:52:41.856338 22904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:52:41.856945 22904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:52:41.857775 22904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:52:41.857848 22904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:41.857923 22904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:52:41.857959 22904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:52:41.867069 22904 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:35257
I20260621 07:52:41.867093 23032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:35257 every 8 connection(s)
I20260621 07:52:41.867950 22904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:52:41.876443 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 22904
I20260621 07:52:41.876626 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:52:41.876940 12336 external_mini_cluster.cc:1428] Setting key 7fd31630b2730fe6e2029f8a57acabb2
Jun 21 07:52:41 dist-test-slave-lbpt krb5kdc[22528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028361, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:52:41.882822 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:41.869826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:43079 (local address 127.12.12.62:39261)
0621 07:52:41.869961 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:41.869965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:41.870792 (+ 827us) server_negotiation.cc:407] Connection header received
0621 07:52:41.872194 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:41.872198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:41.872259 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:41.872340 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:41.873531 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.874113 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.875332 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.875571 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.875688 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:41.879061 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:41.879064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:41.879066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:41.879093 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:41.880566 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:41.881166 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:41.881169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:41.881170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:41.881266 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:41.881646 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:41.881649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:41.881650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:41.881863 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:41.881956 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:41.882584 (+ 628us) server_negotiation.cc:299] Negotiation successful
0621 07:52:41.882696 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:52:41.883776 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:41.870147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:39261 (local address 127.12.12.3:43079)
0621 07:52:41.870633 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:41.870677 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:41.871961 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:41.872519 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:41.872526 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:41.872883 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:52:41.873363 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:41.873375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.874252 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.874256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:41.875194 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:41.875204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.875729 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.875733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:41.875901 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:41.876615 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:52:41.876631 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:52:41.878825 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:52:41.880683 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:41.880688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:41.880702 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:41.881018 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:41.881356 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:52:41.881359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:52:41.881360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:52:41.881521 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:52:41.882019 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:52:41.882026 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:52:41.882293 (+ 267us) client_negotiation.cc:769] Sending connection context
0621 07:52:41.882508 (+ 215us) client_negotiation.cc:240] Negotiation successful
0621 07:52:41.882804 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":114,"threads_started":1}
I20260621 07:52:41.885051 23033 heartbeater.cc:347] Connected to a master server at 127.12.12.62:39261
I20260621 07:52:41.885329 23033 heartbeater.cc:464] Registering TS with master...
I20260621 07:52:41.885833 23033 heartbeater.cc:511] Master 127.12.12.62:39261 requested a full tablet report, sending...
I20260621 07:52:41.886919 22573 ts_manager.cc:194] Registered new tserver with Master: c86771082c084c1f89eb54d1c4dcc2cd (127.12.12.3:35257)
I20260621 07:52:41.887480 22573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:43079
I20260621 07:52:41.891624 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:52:41.901254 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:41.893644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55908 (local address 127.12.12.62:39261)
0621 07:52:41.893776 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:41.893779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:41.893928 (+ 149us) server_negotiation.cc:407] Connection header received
0621 07:52:41.894038 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:41.894041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:41.894079 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:41.894146 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:52:41.894920 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.895514 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.896383 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.896608 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.896698 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:41.898087 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:52:41.898090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:52:41.898092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:52:41.898113 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:52:41.899666 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:41.900078 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:41.900081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:41.900082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:41.900154 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:52:41.900447 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:52:41.900450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:52:41.900452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:52:41.900701 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:52:41.900806 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:41.901044 (+ 238us) server_negotiation.cc:299] Negotiation successful
0621 07:52:41.901137 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:52:41.904359 22573 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:52:41.907662 22573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:52:41.922006 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:41.915870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:39621 (local address 127.0.0.1:49214)
0621 07:52:41.916587 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:41.916640 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:41.916781 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:41.917704 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:41.917707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:41.917723 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:41.917868 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:41.917873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.919445 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.919448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:41.920641 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:41.920652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.921449 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.921452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:41.921557 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:41.921593 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 07:52:41.921704 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 07:52:41.921825 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":70,"spinlock_wait_cycles":5760,"thread_start_us":106,"threads_started":1}
I20260621 07:52:41.922048 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:41.916154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:35257 (local address 127.0.0.1:40298)
0621 07:52:41.916589 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:41.916640 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:41.916781 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:41.917575 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:41.917580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:41.917602 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:41.917783 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:41.917794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.919117 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.919121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:41.920556 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:41.920566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.921691 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.921697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:41.921777 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:41.921795 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 07:52:41.921871 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 07:52:41.921935 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":50,"spinlock_wait_cycles":2688,"thread_start_us":124,"threads_started":1}
I20260621 07:52:41.922184 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:41.916699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40298 (local address 127.12.12.3:35257)
0621 07:52:41.917053 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:41.917064 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:41.917088 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:52:41.917163 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:41.917169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:41.917406 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:41.917929 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:41.918074 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.918982 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.920689 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.921542 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.921684 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:41.921799 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:41.921903 (+ 104us) server_negotiation.cc:299] Negotiation successful
0621 07:52:41.922013 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":190,"threads_started":1}
I20260621 07:52:41.922276 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:41.916245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:41153 (local address 127.0.0.1:48760)
0621 07:52:41.917265 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:41.917278 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:41.917360 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:41.918052 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:41.918056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:41.918074 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:41.918155 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:41.918160 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.919524 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.919527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:41.920584 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:41.920592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.921866 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.921870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:41.921954 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:41.921985 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 07:52:41.922056 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 07:52:41.922130 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":2,"thread_start_us":73,"threads_started":1}
I20260621 07:52:41.922641 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:41.916065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49214 (local address 127.12.12.1:39621)
0621 07:52:41.917238 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:41.917253 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:41.917277 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:52:41.917354 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:41.917359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:41.917546 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:41.917986 (+ 440us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:41.918107 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.919300 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.920738 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.921319 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.921449 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:41.921608 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:41.922311 (+ 703us) server_negotiation.cc:299] Negotiation successful
0621 07:52:41.922462 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1060,"thread_start_us":56,"threads_started":1}
I20260621 07:52:41.922670 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:41.916694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.12.12.2:41153)
0621 07:52:41.917053 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:41.917066 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:41.917456 (+ 390us) server_negotiation.cc:407] Connection header received
0621 07:52:41.917550 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:41.917555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:41.917718 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:41.917870 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:41.918288 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.919398 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.920743 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:41.921541 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:41.921659 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:41.921799 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:41.922372 (+ 573us) server_negotiation.cc:299] Negotiation successful
0621 07:52:41.922513 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":142,"threads_started":1}
I20260621 07:52:41.925541 22967 tablet_service.cc:1467] Processing CreateTablet for tablet 8d611e7e12ee456bb40a789b84870525 (DEFAULT_TABLE table=test-table [id=159b3f3ec2df4de19f5423a79641cb2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:41.925544 22833 tablet_service.cc:1467] Processing CreateTablet for tablet 8d611e7e12ee456bb40a789b84870525 (DEFAULT_TABLE table=test-table [id=159b3f3ec2df4de19f5423a79641cb2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:41.925544 22699 tablet_service.cc:1467] Processing CreateTablet for tablet 8d611e7e12ee456bb40a789b84870525 (DEFAULT_TABLE table=test-table [id=159b3f3ec2df4de19f5423a79641cb2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:52:41.926630 22967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d611e7e12ee456bb40a789b84870525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:41.926632 22833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d611e7e12ee456bb40a789b84870525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:41.926632 22699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d611e7e12ee456bb40a789b84870525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:52:41.933359 23050 tablet_bootstrap.cc:492] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd: Bootstrap starting.
I20260621 07:52:41.934737 23052 tablet_bootstrap.cc:492] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956: Bootstrap starting.
I20260621 07:52:41.934736 23051 tablet_bootstrap.cc:492] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3: Bootstrap starting.
I20260621 07:52:41.935592 23050 tablet_bootstrap.cc:654] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:41.936700 23050 log.cc:824] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:41.936895 23052 tablet_bootstrap.cc:654] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:41.937371 23051 tablet_bootstrap.cc:654] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3: Neither blocks nor log segments found. Creating new log.
I20260621 07:52:41.938280 23052 log.cc:824] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:41.938279 23051 log.cc:824] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3: Log is configured to *not* fsync() on all Append() calls
I20260621 07:52:41.938795 23050 tablet_bootstrap.cc:492] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd: No bootstrap required, opened a new log
I20260621 07:52:41.939002 23050 ts_tablet_manager.cc:1397] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:52:41.940420 23051 tablet_bootstrap.cc:492] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3: No bootstrap required, opened a new log
I20260621 07:52:41.940444 23052 tablet_bootstrap.cc:492] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956: No bootstrap required, opened a new log
I20260621 07:52:41.940613 23052 ts_tablet_manager.cc:1397] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:52:41.940660 23051 ts_tablet_manager.cc:1397] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:52:41.942135 23050 raft_consensus.cc:348] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a503e2b4cf47a99e31681df1868956" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39621 } } peers { permanent_uuid: "c86771082c084c1f89eb54d1c4dcc2cd" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35257 } } peers { permanent_uuid: "7691bcd947414b0bb234e7f7dd92b5c3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 41153 } }
I20260621 07:52:41.942433 23050 raft_consensus.cc:374] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:41.942528 23050 raft_consensus.cc:729] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86771082c084c1f89eb54d1c4dcc2cd, State: Initialized, Role: FOLLOWER
I20260621 07:52:41.943005 23050 consensus_queue.cc:260] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a503e2b4cf47a99e31681df1868956" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39621 } } peers { permanent_uuid: "c86771082c084c1f89eb54d1c4dcc2cd" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35257 } } peers { permanent_uuid: "7691bcd947414b0bb234e7f7dd92b5c3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 41153 } }
I20260621 07:52:41.943799 23033 heartbeater.cc:503] Master 127.12.12.62:39261 was elected leader, sending a full tablet report...
I20260621 07:52:41.944132 23050 ts_tablet_manager.cc:1428] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:52:41.944355 23051 raft_consensus.cc:348] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a503e2b4cf47a99e31681df1868956" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39621 } } peers { permanent_uuid: "c86771082c084c1f89eb54d1c4dcc2cd" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35257 } } peers { permanent_uuid: "7691bcd947414b0bb234e7f7dd92b5c3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 41153 } }
I20260621 07:52:41.944669 23051 raft_consensus.cc:374] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:41.944767 23051 raft_consensus.cc:729] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7691bcd947414b0bb234e7f7dd92b5c3, State: Initialized, Role: FOLLOWER
I20260621 07:52:41.945288 23051 consensus_queue.cc:260] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a503e2b4cf47a99e31681df1868956" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39621 } } peers { permanent_uuid: "c86771082c084c1f89eb54d1c4dcc2cd" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35257 } } peers { permanent_uuid: "7691bcd947414b0bb234e7f7dd92b5c3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 41153 } }
I20260621 07:52:41.945757 23052 raft_consensus.cc:348] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a503e2b4cf47a99e31681df1868956" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39621 } } peers { permanent_uuid: "c86771082c084c1f89eb54d1c4dcc2cd" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35257 } } peers { permanent_uuid: "7691bcd947414b0bb234e7f7dd92b5c3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 41153 } }
I20260621 07:52:41.945930 22899 heartbeater.cc:503] Master 127.12.12.62:39261 was elected leader, sending a full tablet report...
I20260621 07:52:41.946058 23052 raft_consensus.cc:374] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:52:41.946153 23052 raft_consensus.cc:729] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a503e2b4cf47a99e31681df1868956, State: Initialized, Role: FOLLOWER
I20260621 07:52:41.946306 23051 ts_tablet_manager.cc:1428] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260621 07:52:41.946688 23052 consensus_queue.cc:260] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a503e2b4cf47a99e31681df1868956" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39621 } } peers { permanent_uuid: "c86771082c084c1f89eb54d1c4dcc2cd" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35257 } } peers { permanent_uuid: "7691bcd947414b0bb234e7f7dd92b5c3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 41153 } }
I20260621 07:52:41.947672 23052 ts_tablet_manager.cc:1428] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260621 07:52:41.947764 22765 heartbeater.cc:503] Master 127.12.12.62:39261 was elected leader, sending a full tablet report...
W20260621 07:52:42.118930 23034 tablet.cc:2369] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:52:42.129650 23056 raft_consensus.cc:482] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:52:42.129900 23056 raft_consensus.cc:504] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a503e2b4cf47a99e31681df1868956" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39621 } } peers { permanent_uuid: "c86771082c084c1f89eb54d1c4dcc2cd" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35257 } } peers { permanent_uuid: "7691bcd947414b0bb234e7f7dd92b5c3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 41153 } }
I20260621 07:52:42.131227 23056 leader_election.cc:288] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98a503e2b4cf47a99e31681df1868956 (127.12.12.1:39621), 7691bcd947414b0bb234e7f7dd92b5c3 (127.12.12.2:41153)
I20260621 07:52:42.135016 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:42.131566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:41153 (local address 127.12.12.3:41337)
0621 07:52:42.131678 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:42.131695 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:42.131801 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:42.132165 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:42.132168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:42.132186 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:42.132306 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:42.132312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:42.133061 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:42.133066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:42.133876 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:42.133885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:42.134578 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:42.134581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:42.134685 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:42.134713 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 07:52:42.134801 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 07:52:42.134872 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260621 07:52:42.135012 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:42.131721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:41337 (local address 127.12.12.2:41153)
0621 07:52:42.131879 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:42.131886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:42.131903 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:52:42.131988 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:42.131991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:42.132040 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:42.132138 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:42.132427 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:42.132936 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:42.134023 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:42.134450 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:42.134576 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:42.134643 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:42.134815 (+ 172us) server_negotiation.cc:299] Negotiation successful
0621 07:52:42.134890 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:52:42.135447 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:42.131566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:39621 (local address 127.12.12.3:53695)
0621 07:52:42.132246 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:42.132257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:52:42.132381 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:52:42.132656 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:52:42.132659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:52:42.132674 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:52:42.132800 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:42.132805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:42.133557 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:42.133562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:42.134337 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:52:42.134346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:52:42.135018 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:52:42.135021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:52:42.135118 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:42.135146 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 07:52:42.135239 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 07:52:42.135299 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":307,"thread_start_us":142,"threads_started":1}
I20260621 07:52:42.135676 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:42.131806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:53695 (local address 127.12.12.1:39621)
0621 07:52:42.131958 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:42.131963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:42.132353 (+ 390us) server_negotiation.cc:407] Connection header received
0621 07:52:42.132507 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:42.132512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:42.132558 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:42.132645 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:52:42.132931 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:42.133438 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:42.134509 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:42.134899 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:42.135015 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:42.135073 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:42.135461 (+ 388us) server_negotiation.cc:299] Negotiation successful
0621 07:52:42.135553 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:52:42.135809 22853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d611e7e12ee456bb40a789b84870525" candidate_uuid: "c86771082c084c1f89eb54d1c4dcc2cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7691bcd947414b0bb234e7f7dd92b5c3" is_pre_election: true
I20260621 07:52:42.136180 22853 raft_consensus.cc:2454] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86771082c084c1f89eb54d1c4dcc2cd in term 0.
I20260621 07:52:42.136209 22719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d611e7e12ee456bb40a789b84870525" candidate_uuid: "c86771082c084c1f89eb54d1c4dcc2cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a503e2b4cf47a99e31681df1868956" is_pre_election: true
I20260621 07:52:42.136523 22719 raft_consensus.cc:2454] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86771082c084c1f89eb54d1c4dcc2cd in term 0.
I20260621 07:52:42.136685 22922 leader_election.cc:302] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7691bcd947414b0bb234e7f7dd92b5c3, c86771082c084c1f89eb54d1c4dcc2cd; no voters:
I20260621 07:52:42.136987 23056 raft_consensus.cc:2781] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:52:42.137086 23056 raft_consensus.cc:482] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:52:42.137130 23056 raft_consensus.cc:3037] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:42.138177 23056 raft_consensus.cc:504] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a503e2b4cf47a99e31681df1868956" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39621 } } peers { permanent_uuid: "c86771082c084c1f89eb54d1c4dcc2cd" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35257 } } peers { permanent_uuid: "7691bcd947414b0bb234e7f7dd92b5c3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 41153 } }
I20260621 07:52:42.138563 23056 leader_election.cc:288] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [CANDIDATE]: Term 1 election: Requested vote from peers 98a503e2b4cf47a99e31681df1868956 (127.12.12.1:39621), 7691bcd947414b0bb234e7f7dd92b5c3 (127.12.12.2:41153)
I20260621 07:52:42.138906 22719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d611e7e12ee456bb40a789b84870525" candidate_uuid: "c86771082c084c1f89eb54d1c4dcc2cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a503e2b4cf47a99e31681df1868956"
I20260621 07:52:42.138975 22853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d611e7e12ee456bb40a789b84870525" candidate_uuid: "c86771082c084c1f89eb54d1c4dcc2cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7691bcd947414b0bb234e7f7dd92b5c3"
I20260621 07:52:42.139057 22719 raft_consensus.cc:3037] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:42.139128 22853 raft_consensus.cc:3037] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:52:42.140129 22719 raft_consensus.cc:2454] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86771082c084c1f89eb54d1c4dcc2cd in term 1.
I20260621 07:52:42.140265 22853 raft_consensus.cc:2454] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86771082c084c1f89eb54d1c4dcc2cd in term 1.
I20260621 07:52:42.140422 22921 leader_election.cc:302] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98a503e2b4cf47a99e31681df1868956, c86771082c084c1f89eb54d1c4dcc2cd; no voters:
I20260621 07:52:42.140589 23056 raft_consensus.cc:2781] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:52:42.140820 23056 raft_consensus.cc:686] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [term 1 LEADER]: Becoming Leader. State: Replica: c86771082c084c1f89eb54d1c4dcc2cd, State: Running, Role: LEADER
I20260621 07:52:42.141173 23056 consensus_queue.cc:237] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a503e2b4cf47a99e31681df1868956" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39621 } } peers { permanent_uuid: "c86771082c084c1f89eb54d1c4dcc2cd" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35257 } } peers { permanent_uuid: "7691bcd947414b0bb234e7f7dd92b5c3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 41153 } }
W20260621 07:52:42.144824 22900 tablet.cc:2369] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:52:42.144572 22570 catalog_manager.cc:5495] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd reported cstate change: term changed from 0 to 1, leader changed from <none> to c86771082c084c1f89eb54d1c4dcc2cd (127.12.12.3). New cstate: current_term: 1 leader_uuid: "c86771082c084c1f89eb54d1c4dcc2cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a503e2b4cf47a99e31681df1868956" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86771082c084c1f89eb54d1c4dcc2cd" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7691bcd947414b0bb234e7f7dd92b5c3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 41153 } health_report { overall_health: UNKNOWN } } }
I20260621 07:52:42.156675 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:42.153038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.12.12.3:35257)
0621 07:52:42.153163 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 07:52:42.153167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:52:42.153180 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:52:42.153300 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:52:42.153302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:52:42.153357 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:52:42.153448 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:52:42.153718 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:42.154262 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:42.155232 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:52:42.155445 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:52:42.155559 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:52:42.155831 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:52:42.156239 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:52:42.156329 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 07:52:42.156503 (+ 174us) server_negotiation.cc:299] Negotiation successful
0621 07:52:42.156553 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:52:42.162774 22853 raft_consensus.cc:1261] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3 [term 1 FOLLOWER]: Refusing update from remote peer c86771082c084c1f89eb54d1c4dcc2cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:42.162981 22719 raft_consensus.cc:1261] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956 [term 1 FOLLOWER]: Refusing update from remote peer c86771082c084c1f89eb54d1c4dcc2cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:52:42.163633 23060 consensus_queue.cc:1035] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7691bcd947414b0bb234e7f7dd92b5c3" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 41153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:42.163867 23056 consensus_queue.cc:1035] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "98a503e2b4cf47a99e31681df1868956" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:52:42.170786 23063 mvcc.cc:205] Tried to move back new op lower bound from 7299188171411017728 to 7299188171333115904. Current Snapshot: MvccSnapshot[applied={T|T < 7299188171411017728}]
I20260621 07:52:42.172952 23065 mvcc.cc:205] Tried to move back new op lower bound from 7299188171411017728 to 7299188171333115904. Current Snapshot: MvccSnapshot[applied={T|T < 7299188171411017728 or (T in {7299188171411017728})}]
I20260621 07:52:42.174660 23064 mvcc.cc:205] Tried to move back new op lower bound from 7299188171411017728 to 7299188171333115904. Current Snapshot: MvccSnapshot[applied={T|T < 7299188171411017728}]
I20260621 07:52:42.179143 22573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:42.179327 22573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:52:42.181841 22573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8a5c4549f494b39af8b7df64be95704: Sending DeleteTablet for 3 replicas of tablet 8d611e7e12ee456bb40a789b84870525
I20260621 07:52:42.182667 22699 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d611e7e12ee456bb40a789b84870525 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:42 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:49214
I20260621 07:52:42.182668 22967 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d611e7e12ee456bb40a789b84870525 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:42 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:40298
I20260621 07:52:42.182745 22833 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d611e7e12ee456bb40a789b84870525 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:52:42 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:48760
I20260621 07:52:42.183197 23075 tablet_replica.cc:330] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd: stopping tablet replica
I20260621 07:52:42.183442 23074 tablet_replica.cc:330] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956: stopping tablet replica
I20260621 07:52:42.183544 23075 raft_consensus.cc:2229] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:52:42.183708 23074 raft_consensus.cc:2229] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:42.183871 23075 raft_consensus.cc:2258] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:42.184019 23074 raft_consensus.cc:2258] T 8d611e7e12ee456bb40a789b84870525 P 98a503e2b4cf47a99e31681df1868956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:42.184214 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 22636
I20260621 07:52:42.184343 23076 tablet_replica.cc:330] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3: stopping tablet replica
I20260621 07:52:42.184612 23076 raft_consensus.cc:2229] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:52:42.184902 23076 raft_consensus.cc:2258] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:52:42.185428 23075 ts_tablet_manager.cc:1905] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:42.186223 23076 ts_tablet_manager.cc:1905] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:52:42.188342 23075 ts_tablet_manager.cc:1918] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:42.188427 23075 log.cc:1196] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/8d611e7e12ee456bb40a789b84870525
I20260621 07:52:42.188772 23075 ts_tablet_manager.cc:1939] T 8d611e7e12ee456bb40a789b84870525 P c86771082c084c1f89eb54d1c4dcc2cd: Deleting consensus metadata
I20260621 07:52:42.188916 23076 ts_tablet_manager.cc:1918] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:52:42.189049 23076 log.cc:1196] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/8d611e7e12ee456bb40a789b84870525
I20260621 07:52:42.189584 23076 ts_tablet_manager.cc:1939] T 8d611e7e12ee456bb40a789b84870525 P 7691bcd947414b0bb234e7f7dd92b5c3: Deleting consensus metadata
I20260621 07:52:42.190026 22560 catalog_manager.cc:4841] TS c86771082c084c1f89eb54d1c4dcc2cd (127.12.12.3:35257): tablet 8d611e7e12ee456bb40a789b84870525 (table test-table [id=159b3f3ec2df4de19f5423a79641cb2b]) successfully deleted
I20260621 07:52:42.190828 22559 catalog_manager.cc:4841] TS 7691bcd947414b0bb234e7f7dd92b5c3 (127.12.12.2:41153): tablet 8d611e7e12ee456bb40a789b84870525 (table test-table [id=159b3f3ec2df4de19f5423a79641cb2b]) successfully deleted
W20260621 07:52:42.194674 22558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.1:39621 (error 108)
I20260621 07:52:42.195606 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 22770
I20260621 07:52:42.195858 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:52:42.195532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:39621 (local address 127.0.0.1:49230)
0621 07:52:42.195686 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 07:52:42.195757 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:39621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260621 07:52:42.196955 22558 catalog_manager.cc:4584] TS 98a503e2b4cf47a99e31681df1868956 (127.12.12.1:39621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d611e7e12ee456bb40a789b84870525: Network error: Client connection negotiation failed: client connection to 127.12.12.1:39621: connect: Connection refused (error 111)
I20260621 07:52:42.201565 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 22904
I20260621 07:52:42.207710 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 22541
2026-06-21T07:52:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2474 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:52:42 dist-test-slave-lbpt krb5kdc[23080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:52:42 dist-test-slave-lbpt krb5kdc[23080](info): set up 2 sockets
Jun 21 07:52:42 dist-test-slave-lbpt krb5kdc[23080](info): commencing operation
krb5kdc: starting...
W20260621 07:52:43.247846 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:52:43 dist-test-slave-lbpt krb5kdc[23080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:52:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:52:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.12.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.12.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.12.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/rangeradmin_127.12.12.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.12.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/rangeradmin_127.12.12.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.12.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.12.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.12.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/rangerlookup_127.12.12.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.12.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/rangerlookup_127.12.12.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.12.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/HTTP_127.12.12.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.12.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/HTTP_127.12.12.20@KRBTEST.COM.keytab.
I20260621 07:52:43.415469 12336 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskzeuNYm/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/postgres -l logfile start
2026-06-21 07:52:45.927 UTC [23110] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 07:52:45.927 UTC [23110] LOG: listening on IPv4 address "127.12.12.20", port 44615
2026-06-21 07:52:45.930 UTC [23110] LOG: listening on Unix socket "/tmp/.s.PGSQL.44615"
2026-06-21 07:52:45.951 UTC [23113] LOG: database system was shut down at 2026-06-21 07:52:44 UTC
2026-06-21 07:52:45.955 UTC [23110] LOG: database system is ready to accept connections
I20260621 07:52:46.919344 12336 mini_postgres.cc:96] Postgres bound to 44615
2026-06-21 07:52:46.925 UTC [23121] FATAL: database "slave" does not exist
127.12.12.20:44615 - accepting connections
I20260621 07:52:46.926564 12336 mini_ranger.cc:161] Starting Ranger...
I20260621 07:52:46.941483 12336 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 07:52:47.411303 12336 mini_ranger.cc:87] Created ranger Postgres database
I20260621 07:52:47.411417 12336 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-admin
2026-06-21 07:52:47,697 [I] DB FLAVOR :POSTGRES
2026-06-21 07:52:47,698 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 07:52:47,698 [I] Checking connection..
2026-06-21 07:52:47,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 07:52:47,962 [I] Checking connection passed.
2026-06-21 07:52:47,962 [I] --------- Verifying version history table ---------
2026-06-21 07:52:47,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:52:48,234 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 07:52:48,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:52:48,473 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 07:52:48,473 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 07:52:48,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 07:52:48.734 UTC [23185] WARNING: there is no transaction in progress
2026-06-21 07:52:48,740 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 07:52:48,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:52:48,989 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 07:52:48,989 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 07:52:48,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 07:52:49,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 07:52:49,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lbpt.c.gcp-upstream.internal','N') ;"
2026-06-21 07:52:49,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 07:52:49,966 [I] Table x_portal_user does not exist in database ranger
2026-06-21 07:52:49,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 07:52:50,217 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 07:52:50,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 07:52:50,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 07:52:50,705 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 07:52:50,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 07:52:51.638 UTC [23293] WARNING: there is no transaction in progress
2026-06-21 07:52:51.656 UTC [23293] WARNING: there is no transaction in progress
2026-06-21 07:52:51.673 UTC [23293] WARNING: there is no transaction in progress
2026-06-21 07:52:51.688 UTC [23293] WARNING: there is no transaction in progress
2026-06-21 07:52:51.706 UTC [23293] WARNING: there is no transaction in progress
2026-06-21 07:52:51.724 UTC [23293] WARNING: there is no transaction in progress
2026-06-21 07:52:51.788 UTC [23293] WARNING: there is no transaction in progress
2026-06-21 07:52:51.795 UTC [23293] WARNING: there is no transaction in progress
2026-06-21 07:52:51.809 UTC [23293] WARNING: there is no transaction in progress
2026-06-21 07:52:51.819 UTC [23293] WARNING: there is no transaction in progress
2026-06-21 07:52:51.829 UTC [23293] WARNING: there is no transaction in progress
2026-06-21 07:52:51.842 UTC [23293] WARNING: there is no transaction in progress
2026-06-21 07:52:51.852 UTC [23293] WARNING: there is no transaction in progress
2026-06-21 07:52:51.858 UTC [23293] WARNING: there is no transaction in progress
2026-06-21 07:52:51.864 UTC [23293] WARNING: there is no transaction in progress
2026-06-21 07:52:51.871 UTC [23293] WARNING: there is no transaction in progress
2026-06-21 07:52:52,506 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 07:52:52,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 07:52:52,745 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 07:52:52,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 07:52:52,985 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 07:52:52,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 07:52:53,236 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 07:52:53,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 07:52:53,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 07:52:53,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lbpt.c.gcp-upstream.internal';"
2026-06-21 07:52:53,977 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 07:52:53,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 07:52:54,220 [I] DB_PATCHES have already been applied
I20260621 07:52:54.226385 12336 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-admin:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskzeuNYm/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 07:52:54.226536 12336 mini_ranger.cc:193] Using host: 127.12.12.20
I20260621 07:52:54.230134 12336 mini_ranger.cc:227] Ranger admin URL: http://127.12.12.20:37105
Jun 21, 2026 7:52:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 7:52:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 7:52:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 7:52:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 7:52:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 7:52:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 7:52:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 07:52:55.346520 12336 mini_ranger.cc:160] Time spent starting Ranger: real 8.420s user 0.001s sys 0.010s
Jun 21, 2026 7:53:14 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 7:53:14 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 21, 2026 7:53:14 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 7:53:14 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 7:53:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 7:53:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 7:53:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 7:53:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 7:53:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 7:53:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 7:53:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 7:53:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 7:53:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 7:53:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 7:53:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 7:53:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 7:53:14 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 7:53:15 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 07:53:16.291975 12336 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.12.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.12.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.12.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/rangerkms_127.12.12.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.12.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/rangerkms_127.12.12.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.12.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.12.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/HTTP_127.12.12.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.12.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/HTTP_127.12.12.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 07:53:16 dist-test-slave-lbpt krb5kdc[23080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028396, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 07:53:16.377894 12336 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 07:53:16.387825 12336 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 07:53:17.225008 12336 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 07:53:17.225090 12336 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-kms
I20260621 07:53:17.225185 12336 mini_ranger_kms.cc:227] Using postgres at 127.12.12.20:44615
2026-06-21 07:53:17,406 [I] DB FLAVOR :POSTGRES
2026-06-21 07:53:17,407 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 07:53:17,407 [I] Checking connection
2026-06-21 07:53:17,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 07:53:17,659 [I] connection success
2026-06-21 07:53:17,659 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 07:53:17,659 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 07:53:17,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 07:53:17,904 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 07:53:17,905 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 07:53:17,905 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 07:53:17,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:44615/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 07:53:18,164 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 07:53:18.623718 12336 mini_ranger_kms.cc:310] Created kms service
I20260621 07:53:18.822371 12336 mini_ranger_kms.cc:326] Created kudu user
I20260621 07:53:18.870867 12336 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 07:53:18 dist-test-slave-lbpt krb5kdc[23080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028396, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:53:19.294073 12336 mini_ranger_kms.cc:384] Added ranger policy
I20260621 07:53:19.294402 12336 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-kms:/tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 07:53:19.294516 12336 mini_ranger_kms.cc:240] Using host: 127.12.12.20
I20260621 07:53:19.298048 12336 mini_ranger_kms.cc:276] Ranger KMS PID: 23473
I20260621 07:53:19.298194 12336 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.12.20:58147
Jun 21, 2026 7:53:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 7:53:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 7:53:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 7:53:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 7:53:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 7:53:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 21 07:53:21 dist-test-slave-lbpt krb5kdc[23080](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782028401, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.12.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:53:21.586138 12336 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.208s user 0.005s sys 0.160s
I20260621 07:53:21.586279 12336 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 07:53:21.586401 12336 mini_ranger_kms.cc:401] 127.12.12.20:58147/kms/v1/keys
Jun 21 07:53:21 dist-test-slave-lbpt krb5kdc[23080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028401, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.12.20@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
Jun 21 07:53:22 dist-test-slave-lbpt krb5kdc[23080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028401, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.12.20@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
Jun 21 07:53:24 dist-test-slave-lbpt krb5kdc[23080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:53:24.578924 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:46241
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:41205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:46241
--ranger_config_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.20:58147/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:53:24.699510 23524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:53:24.699790 23524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:53:24.699831 23524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:53:24.704013 23524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:53:24.704085 23524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:53:24.704105 23524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:53:24.704121 23524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:53:24.704136 23524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:53:24.710338 23524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:41205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.12.20:58147/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:46241
--ranger_config_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:46241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:53:24.711541 23524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:53:24.712610 23524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:53:24.721740 23529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:53:24.721696 23532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:53:24.721777 23530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:53:24.721982 23524 server_base.cc:1034] running on GCE node
I20260621 07:53:24.722798 23524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:53:24.723976 23524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:53:24.725217 23524 hybrid_clock.cc:648] HybridClock initialized: now 1782028404725192 us; error 35 us; skew 500 ppm
Jun 21 07:53:24 dist-test-slave-lbpt krb5kdc[23080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028404, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:53:24.732142 23524 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:53:24.733258 23524 webserver.cc:460] Webserver started at http://127.12.12.62:33511/ using document root <none> and password file <none>
I20260621 07:53:24.733822 23524 fs_manager.cc:359] Metadata directory not provided
I20260621 07:53:24.733906 23524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:53:24.734143 23524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:53:24 dist-test-slave-lbpt krb5kdc[23080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028404, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:53:24.783067 23524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "bb6e11d7fe394d86a57ea7f47292b5c1"
format_stamp: "Formatted at 2026-06-21 07:53:24 on dist-test-slave-lbpt"
server_key: "51c2211352365d70b540da571c6ccf41"
server_key_iv: "6145f162d88666ea050e97d2f5664e7a"
server_key_version: "kuduclusterkey@0"
I20260621 07:53:24.783783 23524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "bb6e11d7fe394d86a57ea7f47292b5c1"
format_stamp: "Formatted at 2026-06-21 07:53:24 on dist-test-slave-lbpt"
server_key: "51c2211352365d70b540da571c6ccf41"
server_key_iv: "6145f162d88666ea050e97d2f5664e7a"
server_key_version: "kuduclusterkey@0"
I20260621 07:53:24.788877 23524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260621 07:53:24.806849 23539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:53:24.808549 23524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:53:24.808696 23524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "bb6e11d7fe394d86a57ea7f47292b5c1"
format_stamp: "Formatted at 2026-06-21 07:53:24 on dist-test-slave-lbpt"
server_key: "51c2211352365d70b540da571c6ccf41"
server_key_iv: "6145f162d88666ea050e97d2f5664e7a"
server_key_version: "kuduclusterkey@0"
I20260621 07:53:24.808838 23524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:53:24.851608 23524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:53:24.855342 23524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:53:24.855556 23524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:53:24.864499 23524 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:46241
I20260621 07:53:24.865489 23591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:46241 every 8 connection(s)
I20260621 07:53:24.865798 23524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:53:24.868311 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 23524
I20260621 07:53:24.868510 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:53:24.868839 12336 external_mini_cluster.cc:1428] Setting key 7be80b39781c775a9f6af07d3646e56b
I20260621 07:53:24.869530 23592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:53:24.877058 23592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1: Bootstrap starting.
Jun 21 07:53:24 dist-test-slave-lbpt krb5kdc[23080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:53:24.879877 23592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1: Neither blocks nor log segments found. Creating new log.
I20260621 07:53:24.881209 23592 log.cc:824] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1: Log is configured to *not* fsync() on all Append() calls
I20260621 07:53:24.883826 23592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1: No bootstrap required, opened a new log
I20260621 07:53:24.884852 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:24.870549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.12.12.62:46241)
0621 07:53:24.871006 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0621 07:53:24.871033 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0621 07:53:24.871073 (+ 40us) server_negotiation.cc:407] Connection header received
0621 07:53:24.871832 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:53:24.871850 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:53:24.872216 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:53:24.872597 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:53:24.873497 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:24.874678 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:24.875646 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:24.876185 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:24.876374 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:24.879415 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:53:24.879420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:53:24.879435 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:53:24.879468 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:53:24.881398 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:53:24.882005 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:53:24.882011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:53:24.882018 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:53:24.882166 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:53:24.882535 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:53:24.882538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:53:24.882540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:53:24.883060 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:53:24.883247 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0621 07:53:24.883596 (+ 349us) server_negotiation.cc:299] Negotiation successful
0621 07:53:24.883881 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":148,"threads_started":1}
I20260621 07:53:24.887921 23592 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6e11d7fe394d86a57ea7f47292b5c1" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46241 } }
I20260621 07:53:24.888206 23592 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:53:24.888311 23592 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6e11d7fe394d86a57ea7f47292b5c1, State: Initialized, Role: FOLLOWER
I20260621 07:53:24.888865 23592 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6e11d7fe394d86a57ea7f47292b5c1" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46241 } }
I20260621 07:53:24.889046 23592 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:53:24.889202 23592 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:53:24.889338 23592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:53:24.890544 23592 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6e11d7fe394d86a57ea7f47292b5c1" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46241 } }
I20260621 07:53:24.890930 23592 leader_election.cc:302] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb6e11d7fe394d86a57ea7f47292b5c1; no voters:
I20260621 07:53:24.891294 23592 leader_election.cc:288] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:53:24.891417 23597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:53:24.891669 23597 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [term 1 LEADER]: Becoming Leader. State: Replica: bb6e11d7fe394d86a57ea7f47292b5c1, State: Running, Role: LEADER
I20260621 07:53:24.892055 23597 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6e11d7fe394d86a57ea7f47292b5c1" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46241 } }
I20260621 07:53:24.892740 23592 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:53:24.893769 23598 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb6e11d7fe394d86a57ea7f47292b5c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6e11d7fe394d86a57ea7f47292b5c1" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46241 } } }
I20260621 07:53:24.893885 23598 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [sys.catalog]: This master's current role is: LEADER
I20260621 07:53:24.894102 23598 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb6e11d7fe394d86a57ea7f47292b5c1. Latest consensus state: current_term: 1 leader_uuid: "bb6e11d7fe394d86a57ea7f47292b5c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6e11d7fe394d86a57ea7f47292b5c1" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46241 } } }
I20260621 07:53:24.894165 23598 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1 [sys.catalog]: This master's current role is: LEADER
I20260621 07:53:24.895200 23592 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 07:53:24.899158 23606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:53:24.903550 23606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:53:24.910830 23606 catalog_manager.cc:1347] Generated new cluster ID: 6228a7700ea24302bdfd51bd4c113d36
I20260621 07:53:24.910894 23606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:53:24.922663 23606 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:53:24.924101 23606 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:53:24.934994 23606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1: Generated new TSK 0
I20260621 07:53:24.936077 23606 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:53:25.663341 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.20:58147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:46241
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:41205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:53:25.817692 23637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:53:25.818020 23637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:53:25.818132 23637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:53:25.823297 23637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:53:25.823400 23637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:53:25.823509 23637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:53:25.831655 23637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:41205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.12.20:58147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:46241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:53:25.833137 23637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:53:25.834529 23637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:53:25.846892 23647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:53:25.846904 23645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:53:25.847086 23644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:53:25.847548 23637 server_base.cc:1034] running on GCE node
I20260621 07:53:25.848037 23637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:53:25.848860 23637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:53:25.850035 23637 hybrid_clock.cc:648] HybridClock initialized: now 1782028405850011 us; error 49 us; skew 500 ppm
Jun 21 07:53:25 dist-test-slave-lbpt krb5kdc[23080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028405, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:53:25.854017 23637 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:53:25.855273 23637 webserver.cc:460] Webserver started at http://127.12.12.1:36263/ using document root <none> and password file <none>
I20260621 07:53:25.856073 23637 fs_manager.cc:359] Metadata directory not provided
I20260621 07:53:25.856240 23637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:53:25.856556 23637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:53:25 dist-test-slave-lbpt krb5kdc[23080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028405, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:53:25.894461 23637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "ef5f47e50ed54f6e91d2473a7ec749d3"
format_stamp: "Formatted at 2026-06-21 07:53:25 on dist-test-slave-lbpt"
server_key: "4029f347d4651da015a627ba44baa211"
server_key_iv: "aa165b07af7cb564bb7f28c7218b9f91"
server_key_version: "kuduclusterkey@0"
I20260621 07:53:25.895397 23637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "ef5f47e50ed54f6e91d2473a7ec749d3"
format_stamp: "Formatted at 2026-06-21 07:53:25 on dist-test-slave-lbpt"
server_key: "4029f347d4651da015a627ba44baa211"
server_key_iv: "aa165b07af7cb564bb7f28c7218b9f91"
server_key_version: "kuduclusterkey@0"
I20260621 07:53:25.901207 23637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 07:53:25.925436 23656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:53:25.927117 23637 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260621 07:53:25.927295 23637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "ef5f47e50ed54f6e91d2473a7ec749d3"
format_stamp: "Formatted at 2026-06-21 07:53:25 on dist-test-slave-lbpt"
server_key: "4029f347d4651da015a627ba44baa211"
server_key_iv: "aa165b07af7cb564bb7f28c7218b9f91"
server_key_version: "kuduclusterkey@0"
I20260621 07:53:25.927610 23637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:53:25.941457 23637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:53:25.944737 23637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:53:25.944940 23637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:53:25.945588 23637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:53:25.946535 23637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:53:25.946604 23637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:53:25.946694 23637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:53:25.946729 23637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:53:25.957726 23637 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:43193
I20260621 07:53:25.958868 23637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:53:25.964440 23769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:43193 every 8 connection(s)
I20260621 07:53:25.964715 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 23637
I20260621 07:53:25.964854 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:53:25.965184 12336 external_mini_cluster.cc:1428] Setting key 6a03d96dfe4f378a3f8c0d906e90883b
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
Jun 21 07:53:25 dist-test-slave-lbpt krb5kdc[23080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028405, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
Jun 21 07:53:25 dist-test-slave-lbpt krb5kdc[23080](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782028405, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:53:25.988787 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:25.968673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46241 (local address 127.12.12.1:47485)
0621 07:53:25.969940 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0621 07:53:25.970023 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0621 07:53:25.971359 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:53:25.972374 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:53:25.972387 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:53:25.973214 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:53:25.974007 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:53:25.974033 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:53:25.975408 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:53:25.975415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:53:25.976921 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:53:25.976934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:53:25.977831 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:53:25.977837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:53:25.978059 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:25.978812 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:53:25.978829 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:53:25.982262 (+ 3433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:53:25.984779 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:53:25.984788 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:53:25.984810 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:53:25.985288 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:53:25.985636 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:53:25.985641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:53:25.985646 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:53:25.985844 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:53:25.986388 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:53:25.986398 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:53:25.986805 (+ 407us) client_negotiation.cc:769] Sending connection context
0621 07:53:25.987053 (+ 248us) client_negotiation.cc:240] Negotiation successful
0621 07:53:25.987473 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"thread_start_us":175,"threads_started":1}
I20260621 07:53:25.989667 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:25.968129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:47485 (local address 127.12.12.62:46241)
0621 07:53:25.971816 (+ 3687us) server_negotiation.cc:206] Beginning negotiation
0621 07:53:25.971823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:53:25.971840 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:53:25.971887 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:53:25.971892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:53:25.971949 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:53:25.972049 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:53:25.974206 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:25.975236 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:25.977087 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:25.977494 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:25.977621 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:25.982547 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:53:25.982552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:53:25.982555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:53:25.982588 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:53:25.984620 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:53:25.985419 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:53:25.985425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:53:25.985429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:53:25.985528 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:53:25.985953 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:53:25.985972 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:53:25.985976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:53:25.986235 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:53:25.986332 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 07:53:25.989315 (+ 2983us) server_negotiation.cc:299] Negotiation successful
0621 07:53:25.989473 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3583,"thread_start_us":86,"threads_started":1}
I20260621 07:53:25.991765 23770 heartbeater.cc:347] Connected to a master server at 127.12.12.62:46241
I20260621 07:53:25.992313 23770 heartbeater.cc:464] Registering TS with master...
I20260621 07:53:25.993373 23770 heartbeater.cc:511] Master 127.12.12.62:46241 requested a full tablet report, sending...
I20260621 07:53:25.996656 23556 ts_manager.cc:194] Registered new tserver with Master: ef5f47e50ed54f6e91d2473a7ec749d3 (127.12.12.1:43193)
I20260621 07:53:25.998720 23556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:47485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:53:26.010699 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.20:58147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:46241
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:41205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:53:26.143424 23776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:53:26.143850 23776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:53:26.143983 23776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:53:26.149112 23776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:53:26.149195 23776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:53:26.149298 23776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:53:26.155424 23776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:41205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.12.20:58147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:46241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:53:26.156819 23776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:53:26.157991 23776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:53:26.170396 23785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:53:26.170678 23783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:53:26.170796 23782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:53:26.171278 23776 server_base.cc:1034] running on GCE node
I20260621 07:53:26.171689 23776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:53:26.172407 23776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:53:26.173584 23776 hybrid_clock.cc:648] HybridClock initialized: now 1782028406173552 us; error 44 us; skew 500 ppm
Jun 21 07:53:26 dist-test-slave-lbpt krb5kdc[23080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028406, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:53:26.176367 23776 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:53:26.177368 23776 webserver.cc:460] Webserver started at http://127.12.12.2:46559/ using document root <none> and password file <none>
I20260621 07:53:26.177999 23776 fs_manager.cc:359] Metadata directory not provided
I20260621 07:53:26.178092 23776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:53:26.178303 23776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:53:26 dist-test-slave-lbpt krb5kdc[23080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028406, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:53:26.194979 23776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "f8f30b1c378e4571a9345eda4f65be54"
format_stamp: "Formatted at 2026-06-21 07:53:26 on dist-test-slave-lbpt"
server_key: "f982a6900a7812515f710b127f31b28c"
server_key_iv: "b951e1748352d70d5c136c4d6f41d871"
server_key_version: "kuduclusterkey@0"
I20260621 07:53:26.195633 23776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "f8f30b1c378e4571a9345eda4f65be54"
format_stamp: "Formatted at 2026-06-21 07:53:26 on dist-test-slave-lbpt"
server_key: "f982a6900a7812515f710b127f31b28c"
server_key_iv: "b951e1748352d70d5c136c4d6f41d871"
server_key_version: "kuduclusterkey@0"
I20260621 07:53:26.199604 23776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:53:26.223495 23792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:53:26.225255 23776 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:53:26.225399 23776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "f8f30b1c378e4571a9345eda4f65be54"
format_stamp: "Formatted at 2026-06-21 07:53:26 on dist-test-slave-lbpt"
server_key: "f982a6900a7812515f710b127f31b28c"
server_key_iv: "b951e1748352d70d5c136c4d6f41d871"
server_key_version: "kuduclusterkey@0"
I20260621 07:53:26.225538 23776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:53:26.244241 23776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:53:26.248076 23776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:53:26.248271 23776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:53:26.248984 23776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:53:26.249998 23776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:53:26.250077 23776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:53:26.250144 23776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:53:26.250177 23776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:53:26.260752 23776 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:46641
I20260621 07:53:26.260914 23905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:46641 every 8 connection(s)
I20260621 07:53:26.261988 23776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:53:26.269335 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 23776
I20260621 07:53:26.269474 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:53:26.269773 12336 external_mini_cluster.cc:1428] Setting key d3a88cba2052387b755b2138551b98a6
Jun 21 07:53:26 dist-test-slave-lbpt krb5kdc[23080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028406, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:53:26.280479 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:26.264565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:46299 (local address 127.12.12.62:46241)
0621 07:53:26.264714 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0621 07:53:26.264719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:53:26.265897 (+ 1178us) server_negotiation.cc:407] Connection header received
0621 07:53:26.267259 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:53:26.267262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:53:26.267320 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:53:26.267423 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:53:26.268850 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.269663 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.271809 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.272174 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.272254 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:26.275494 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:53:26.275498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:53:26.275501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:53:26.275536 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:53:26.277382 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:53:26.278082 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:53:26.278086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:53:26.278087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:53:26.278180 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:53:26.278608 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:53:26.278614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:53:26.278615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:53:26.278857 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:53:26.278967 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 07:53:26.280080 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0621 07:53:26.280240 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:53:26.281108 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:26.265136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46241 (local address 127.12.12.2:46299)
0621 07:53:26.265719 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0621 07:53:26.265783 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0621 07:53:26.266977 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:53:26.267625 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:53:26.267635 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:53:26.267991 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:53:26.268666 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:53:26.268686 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.269822 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.269826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:53:26.271669 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:53:26.271680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.272465 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.272469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:53:26.272689 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:26.273433 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:53:26.273455 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:53:26.275263 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:53:26.277529 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:53:26.277536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:53:26.277557 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:53:26.277953 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:53:26.278280 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:53:26.278283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:53:26.278285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:53:26.278511 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:53:26.278982 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:53:26.278990 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:53:26.279373 (+ 383us) client_negotiation.cc:769] Sending connection context
0621 07:53:26.279593 (+ 220us) client_negotiation.cc:240] Negotiation successful
0621 07:53:26.279963 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":156,"threads_started":1}
I20260621 07:53:26.282541 23906 heartbeater.cc:347] Connected to a master server at 127.12.12.62:46241
I20260621 07:53:26.282809 23906 heartbeater.cc:464] Registering TS with master...
I20260621 07:53:26.283440 23906 heartbeater.cc:511] Master 127.12.12.62:46241 requested a full tablet report, sending...
I20260621 07:53:26.284725 23556 ts_manager.cc:194] Registered new tserver with Master: f8f30b1c378e4571a9345eda4f65be54 (127.12.12.2:46641)
I20260621 07:53:26.285620 23556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:46299
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:53:26.319984 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.20:58147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:46241
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:41205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:53:26 dist-test-slave-lbpt krb5kdc[23080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028405, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
W20260621 07:53:26.518497 23911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:53:26.518903 23911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:53:26.518966 23911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:53:26.534922 23911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:53:26.535017 23911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:53:26.535112 23911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:53:26.553689 23911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:41205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.12.20:58147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:46241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:53:26.555173 23911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:53:26.556537 23911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:53:26.570166 23916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:53:26.570526 23911 server_base.cc:1034] running on GCE node
W20260621 07:53:26.570645 23917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:53:26.570868 23919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:53:26.571342 23911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:53:26.572149 23911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:53:26.573310 23911 hybrid_clock.cc:648] HybridClock initialized: now 1782028406573239 us; error 87 us; skew 500 ppm
Jun 21 07:53:26 dist-test-slave-lbpt krb5kdc[23080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028406, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:53:26.577297 23911 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:53:26.578596 23911 webserver.cc:460] Webserver started at http://127.12.12.3:38511/ using document root <none> and password file <none>
I20260621 07:53:26.579399 23911 fs_manager.cc:359] Metadata directory not provided
I20260621 07:53:26.579486 23911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:53:26.579727 23911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:53:26 dist-test-slave-lbpt krb5kdc[23080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028406, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:53:26.600819 23911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "96c0adedb35847d6984271ab7bf8b64f"
format_stamp: "Formatted at 2026-06-21 07:53:26 on dist-test-slave-lbpt"
server_key: "33cf62bb250f06099aff4eb4d126d3d9"
server_key_iv: "8c0b790064be6cb28a4f1521e7da9d3e"
server_key_version: "kuduclusterkey@0"
I20260621 07:53:26.601619 23911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "96c0adedb35847d6984271ab7bf8b64f"
format_stamp: "Formatted at 2026-06-21 07:53:26 on dist-test-slave-lbpt"
server_key: "33cf62bb250f06099aff4eb4d126d3d9"
server_key_iv: "8c0b790064be6cb28a4f1521e7da9d3e"
server_key_version: "kuduclusterkey@0"
I20260621 07:53:26.616307 23911 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.005s sys 0.002s
I20260621 07:53:26.635777 23926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:53:26.637856 23911 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260621 07:53:26.638012 23911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "96c0adedb35847d6984271ab7bf8b64f"
format_stamp: "Formatted at 2026-06-21 07:53:26 on dist-test-slave-lbpt"
server_key: "33cf62bb250f06099aff4eb4d126d3d9"
server_key_iv: "8c0b790064be6cb28a4f1521e7da9d3e"
server_key_version: "kuduclusterkey@0"
I20260621 07:53:26.638633 23911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:53:26.667963 23911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:53:26.671823 23911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:53:26.672089 23911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:53:26.672921 23911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:53:26.674274 23911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:53:26.674407 23911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:53:26.674593 23911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:53:26.674667 23911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:53:26.690992 23911 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:38869
I20260621 07:53:26.692339 23911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:53:26.693437 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 23911
I20260621 07:53:26.693553 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:53:26.693845 12336 external_mini_cluster.cc:1428] Setting key 19e548910f252c23b0d5649efb0cf9f3
I20260621 07:53:26.693020 24039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:38869 every 8 connection(s)
Jun 21 07:53:26 dist-test-slave-lbpt krb5kdc[23080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028406, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:53:26.717391 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:26.699902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:36869 (local address 127.12.12.62:46241)
0621 07:53:26.700032 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0621 07:53:26.700037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:53:26.700605 (+ 568us) server_negotiation.cc:407] Connection header received
0621 07:53:26.701864 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:53:26.701868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:53:26.701922 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:53:26.701976 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:53:26.704427 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.705268 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.708836 (+ 3568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.709199 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.709286 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:26.712651 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:53:26.712655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:53:26.712658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:53:26.712688 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:53:26.714618 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:53:26.715405 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:53:26.715409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:53:26.715411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:53:26.715502 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:53:26.715964 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:53:26.715967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:53:26.715969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:53:26.716225 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:53:26.716317 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 07:53:26.717071 (+ 754us) server_negotiation.cc:299] Negotiation successful
0621 07:53:26.717209 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:53:26.721323 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:26.698564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46241 (local address 127.12.12.3:36869)
0621 07:53:26.700438 (+ 1874us) negotiation.cc:107] Waiting for socket to connect
0621 07:53:26.700495 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 07:53:26.701620 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:53:26.702394 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:53:26.702404 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:53:26.702864 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:53:26.703985 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:53:26.704007 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.705434 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.705437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:53:26.708683 (+ 3246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:53:26.708697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.709520 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.709537 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:53:26.709770 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:26.710531 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:53:26.710557 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:53:26.712420 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:53:26.714812 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:53:26.714819 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:53:26.714840 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:53:26.715281 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:53:26.715643 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:53:26.715646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:53:26.715648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:53:26.715848 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:53:26.716394 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:53:26.716402 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:53:26.716779 (+ 377us) client_negotiation.cc:769] Sending connection context
0621 07:53:26.717906 (+ 1127us) client_negotiation.cc:240] Negotiation successful
0621 07:53:26.720106 (+ 2200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1646,"thread_start_us":155,"threads_started":1}
I20260621 07:53:26.722973 24040 heartbeater.cc:347] Connected to a master server at 127.12.12.62:46241
I20260621 07:53:26.723246 24040 heartbeater.cc:464] Registering TS with master...
I20260621 07:53:26.723875 24040 heartbeater.cc:511] Master 127.12.12.62:46241 requested a full tablet report, sending...
I20260621 07:53:26.725265 23556 ts_manager.cc:194] Registered new tserver with Master: 96c0adedb35847d6984271ab7bf8b64f (127.12.12.3:38869)
I20260621 07:53:26.726059 23556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:36869
I20260621 07:53:26.731621 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:53:26.745223 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:26.734268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.12.12.62:46241)
0621 07:53:26.734393 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 07:53:26.734400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:53:26.735174 (+ 774us) server_negotiation.cc:407] Connection header received
0621 07:53:26.735396 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:53:26.735400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:53:26.735453 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:53:26.735509 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:53:26.736622 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.737445 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.738718 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.739063 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.739150 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:26.741130 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:53:26.741134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:53:26.741137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:53:26.741164 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:53:26.743087 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:53:26.743774 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:53:26.743777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:53:26.743779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:53:26.743868 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:53:26.744193 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:53:26.744198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:53:26.744202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:53:26.744425 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:53:26.744541 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0621 07:53:26.744911 (+ 370us) server_negotiation.cc:299] Negotiation successful
0621 07:53:26.745042 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:53:26.749660 23556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:53:26.754460 23556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:53:26.785770 24053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:26.769672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:46641 (local address 127.0.0.1:43808)
0621 07:53:26.773972 (+ 4300us) negotiation.cc:107] Waiting for socket to connect
0621 07:53:26.773986 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:53:26.774080 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:53:26.776371 (+ 2291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:53:26.776374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:53:26.776390 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:53:26.776469 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:53:26.776475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.782542 (+ 6067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.782545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:53:26.783711 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:53:26.783722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.785153 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.785157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:53:26.785288 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:26.785330 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 07:53:26.785449 (+ 119us) client_negotiation.cc:240] Negotiation successful
0621 07:53:26.785601 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4237,"thread_start_us":89,"threads_started":1}
I20260621 07:53:26.788796 24055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:26.767064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:38869 (local address 127.0.0.1:41290)
0621 07:53:26.772729 (+ 5665us) negotiation.cc:107] Waiting for socket to connect
0621 07:53:26.772797 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0621 07:53:26.772930 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:53:26.778361 (+ 5431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:53:26.778366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:53:26.778389 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:53:26.778537 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:53:26.778545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.780044 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.780047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:53:26.781142 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:53:26.781154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.788370 (+ 7216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.788374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:53:26.788467 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:26.788488 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:53:26.788573 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 07:53:26.788648 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5445,"thread_start_us":102,"threads_started":1}
I20260621 07:53:26.789754 24057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:26.770856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:43193 (local address 127.0.0.1:50934)
0621 07:53:26.774550 (+ 3694us) negotiation.cc:107] Waiting for socket to connect
0621 07:53:26.774565 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:53:26.774654 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:53:26.775466 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:53:26.775473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:53:26.775493 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:53:26.775632 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:53:26.775642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.777040 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.777044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:53:26.778322 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:53:26.778332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.789344 (+ 11012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.789347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:53:26.789464 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:26.789497 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 07:53:26.789566 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 07:53:26.789637 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3630,"thread_start_us":78,"threads_started":1}
I20260621 07:53:26.789899 24056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:26.770063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43808 (local address 127.12.12.2:46641)
0621 07:53:26.773646 (+ 3583us) server_negotiation.cc:206] Beginning negotiation
0621 07:53:26.773664 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:53:26.774184 (+ 520us) server_negotiation.cc:407] Connection header received
0621 07:53:26.775897 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:53:26.775903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:53:26.776084 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:53:26.776232 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:53:26.776602 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.782352 (+ 5750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.783862 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.784798 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.784908 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:26.785087 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0621 07:53:26.789616 (+ 4529us) server_negotiation.cc:299] Negotiation successful
0621 07:53:26.789735 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3452,"thread_start_us":105,"threads_started":1}
I20260621 07:53:26.790313 24054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:26.768088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.12.12.3:38869)
0621 07:53:26.775284 (+ 7196us) server_negotiation.cc:206] Beginning negotiation
0621 07:53:26.775296 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:53:26.775317 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:53:26.775386 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:53:26.775390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:53:26.775592 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:53:26.775749 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:53:26.778671 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.779903 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.787110 (+ 7207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.788033 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.788147 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:26.788306 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 07:53:26.790048 (+ 1742us) server_negotiation.cc:299] Negotiation successful
0621 07:53:26.790175 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7084,"thread_start_us":149,"threads_started":1}
I20260621 07:53:26.790545 24058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:26.771218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50934 (local address 127.12.12.1:43193)
0621 07:53:26.773837 (+ 2619us) server_negotiation.cc:206] Beginning negotiation
0621 07:53:26.773843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:53:26.774755 (+ 912us) server_negotiation.cc:407] Connection header received
0621 07:53:26.774839 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:53:26.774845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:53:26.775123 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:53:26.775283 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:53:26.775766 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.776873 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.788135 (+ 11262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.788993 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.789120 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:26.789285 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0621 07:53:26.790247 (+ 962us) server_negotiation.cc:299] Negotiation successful
0621 07:53:26.790350 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2523,"thread_start_us":80,"threads_started":1}
I20260621 07:53:26.794643 23700 tablet_service.cc:1467] Processing CreateTablet for tablet 07376db29096403ba41c0498caea36d5 (DEFAULT_TABLE table=test-table [id=fce9b4b9f9804d9b9df9c700f2df1a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:53:26.795900 23700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07376db29096403ba41c0498caea36d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:53:26.795759 23823 tablet_service.cc:1467] Processing CreateTablet for tablet 07376db29096403ba41c0498caea36d5 (DEFAULT_TABLE table=test-table [id=fce9b4b9f9804d9b9df9c700f2df1a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:53:26.796811 23823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07376db29096403ba41c0498caea36d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:53:26.808733 23973 tablet_service.cc:1467] Processing CreateTablet for tablet 07376db29096403ba41c0498caea36d5 (DEFAULT_TABLE table=test-table [id=fce9b4b9f9804d9b9df9c700f2df1a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:53:26.809998 23973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07376db29096403ba41c0498caea36d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:53:26.822072 24060 tablet_bootstrap.cc:492] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54: Bootstrap starting.
I20260621 07:53:26.822817 24059 tablet_bootstrap.cc:492] T 07376db29096403ba41c0498caea36d5 P ef5f47e50ed54f6e91d2473a7ec749d3: Bootstrap starting.
I20260621 07:53:26.824733 24060 tablet_bootstrap.cc:654] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54: Neither blocks nor log segments found. Creating new log.
I20260621 07:53:26.825418 24059 tablet_bootstrap.cc:654] T 07376db29096403ba41c0498caea36d5 P ef5f47e50ed54f6e91d2473a7ec749d3: Neither blocks nor log segments found. Creating new log.
I20260621 07:53:26.825678 24060 log.cc:824] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54: Log is configured to *not* fsync() on all Append() calls
I20260621 07:53:26.826293 24059 log.cc:824] T 07376db29096403ba41c0498caea36d5 P ef5f47e50ed54f6e91d2473a7ec749d3: Log is configured to *not* fsync() on all Append() calls
I20260621 07:53:26.826472 24061 tablet_bootstrap.cc:492] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f: Bootstrap starting.
I20260621 07:53:26.828617 24059 tablet_bootstrap.cc:492] T 07376db29096403ba41c0498caea36d5 P ef5f47e50ed54f6e91d2473a7ec749d3: No bootstrap required, opened a new log
I20260621 07:53:26.828826 24059 ts_tablet_manager.cc:1397] T 07376db29096403ba41c0498caea36d5 P ef5f47e50ed54f6e91d2473a7ec749d3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260621 07:53:26.829011 24061 tablet_bootstrap.cc:654] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f: Neither blocks nor log segments found. Creating new log.
I20260621 07:53:26.830067 24061 log.cc:824] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:53:26.832830 24060 tablet_bootstrap.cc:492] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54: No bootstrap required, opened a new log
I20260621 07:53:26.833035 24060 ts_tablet_manager.cc:1397] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260621 07:53:26.833544 24059 raft_consensus.cc:348] T 07376db29096403ba41c0498caea36d5 P ef5f47e50ed54f6e91d2473a7ec749d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c0adedb35847d6984271ab7bf8b64f" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 38869 } } peers { permanent_uuid: "f8f30b1c378e4571a9345eda4f65be54" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46641 } } peers { permanent_uuid: "ef5f47e50ed54f6e91d2473a7ec749d3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43193 } }
I20260621 07:53:26.833846 24059 raft_consensus.cc:374] T 07376db29096403ba41c0498caea36d5 P ef5f47e50ed54f6e91d2473a7ec749d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:53:26.833923 24059 raft_consensus.cc:729] T 07376db29096403ba41c0498caea36d5 P ef5f47e50ed54f6e91d2473a7ec749d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5f47e50ed54f6e91d2473a7ec749d3, State: Initialized, Role: FOLLOWER
I20260621 07:53:26.834498 24059 consensus_queue.cc:260] T 07376db29096403ba41c0498caea36d5 P ef5f47e50ed54f6e91d2473a7ec749d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c0adedb35847d6984271ab7bf8b64f" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 38869 } } peers { permanent_uuid: "f8f30b1c378e4571a9345eda4f65be54" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46641 } } peers { permanent_uuid: "ef5f47e50ed54f6e91d2473a7ec749d3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43193 } }
I20260621 07:53:26.835597 24059 ts_tablet_manager.cc:1428] T 07376db29096403ba41c0498caea36d5 P ef5f47e50ed54f6e91d2473a7ec749d3: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260621 07:53:26.839067 24060 raft_consensus.cc:348] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c0adedb35847d6984271ab7bf8b64f" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 38869 } } peers { permanent_uuid: "f8f30b1c378e4571a9345eda4f65be54" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46641 } } peers { permanent_uuid: "ef5f47e50ed54f6e91d2473a7ec749d3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43193 } }
I20260621 07:53:26.839363 24060 raft_consensus.cc:374] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:53:26.839421 24060 raft_consensus.cc:729] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f30b1c378e4571a9345eda4f65be54, State: Initialized, Role: FOLLOWER
I20260621 07:53:26.839996 24060 consensus_queue.cc:260] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c0adedb35847d6984271ab7bf8b64f" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 38869 } } peers { permanent_uuid: "f8f30b1c378e4571a9345eda4f65be54" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46641 } } peers { permanent_uuid: "ef5f47e50ed54f6e91d2473a7ec749d3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43193 } }
I20260621 07:53:26.841168 24060 ts_tablet_manager.cc:1428] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260621 07:53:26.841557 23906 heartbeater.cc:503] Master 127.12.12.62:46241 was elected leader, sending a full tablet report...
I20260621 07:53:26.842614 23770 heartbeater.cc:503] Master 127.12.12.62:46241 was elected leader, sending a full tablet report...
I20260621 07:53:26.852169 24061 tablet_bootstrap.cc:492] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f: No bootstrap required, opened a new log
I20260621 07:53:26.852428 24061 ts_tablet_manager.cc:1397] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20260621 07:53:26.856817 24061 raft_consensus.cc:348] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c0adedb35847d6984271ab7bf8b64f" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 38869 } } peers { permanent_uuid: "f8f30b1c378e4571a9345eda4f65be54" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46641 } } peers { permanent_uuid: "ef5f47e50ed54f6e91d2473a7ec749d3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43193 } }
I20260621 07:53:26.857180 24061 raft_consensus.cc:374] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:53:26.857237 24061 raft_consensus.cc:729] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96c0adedb35847d6984271ab7bf8b64f, State: Initialized, Role: FOLLOWER
I20260621 07:53:26.857762 24061 consensus_queue.cc:260] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c0adedb35847d6984271ab7bf8b64f" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 38869 } } peers { permanent_uuid: "f8f30b1c378e4571a9345eda4f65be54" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46641 } } peers { permanent_uuid: "ef5f47e50ed54f6e91d2473a7ec749d3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43193 } }
I20260621 07:53:26.858955 24061 ts_tablet_manager.cc:1428] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260621 07:53:26.859383 24040 heartbeater.cc:503] Master 127.12.12.62:46241 was elected leader, sending a full tablet report...
I20260621 07:53:26.881372 24067 raft_consensus.cc:482] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:53:26.881649 24067 raft_consensus.cc:504] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c0adedb35847d6984271ab7bf8b64f" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 38869 } } peers { permanent_uuid: "f8f30b1c378e4571a9345eda4f65be54" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46641 } } peers { permanent_uuid: "ef5f47e50ed54f6e91d2473a7ec749d3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43193 } }
I20260621 07:53:26.888579 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:26.883517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:46641 (local address 127.12.12.3:40251)
0621 07:53:26.883829 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0621 07:53:26.883847 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:53:26.883958 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:53:26.884587 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:53:26.884591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:53:26.884607 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:53:26.884719 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:53:26.884726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.885735 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.885738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:53:26.887124 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:53:26.887136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.888140 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.888144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:53:26.888255 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:26.888277 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:53:26.888350 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 07:53:26.888426 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260621 07:53:26.889079 24056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:26.884129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:40251 (local address 127.12.12.2:46641)
0621 07:53:26.884249 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0621 07:53:26.884253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:53:26.884268 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:53:26.884321 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:53:26.884324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:53:26.884372 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:53:26.884458 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:53:26.884856 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.885593 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.887276 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.887903 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.888011 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:26.888079 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 07:53:26.888873 (+ 794us) server_negotiation.cc:299] Negotiation successful
0621 07:53:26.888953 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:53:26.889729 23860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07376db29096403ba41c0498caea36d5" candidate_uuid: "96c0adedb35847d6984271ab7bf8b64f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f30b1c378e4571a9345eda4f65be54" is_pre_election: true
I20260621 07:53:26.890082 23860 raft_consensus.cc:2454] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96c0adedb35847d6984271ab7bf8b64f in term 0.
I20260621 07:53:26.890733 23928 leader_election.cc:302] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96c0adedb35847d6984271ab7bf8b64f, f8f30b1c378e4571a9345eda4f65be54; no voters:
I20260621 07:53:26.891939 24067 leader_election.cc:288] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8f30b1c378e4571a9345eda4f65be54 (127.12.12.2:46641), ef5f47e50ed54f6e91d2473a7ec749d3 (127.12.12.1:43193)
I20260621 07:53:26.892057 24067 raft_consensus.cc:2781] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:53:26.892127 24067 raft_consensus.cc:482] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:53:26.892179 24067 raft_consensus.cc:3037] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:53:26.894832 24067 raft_consensus.cc:504] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c0adedb35847d6984271ab7bf8b64f" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 38869 } } peers { permanent_uuid: "f8f30b1c378e4571a9345eda4f65be54" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46641 } } peers { permanent_uuid: "ef5f47e50ed54f6e91d2473a7ec749d3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43193 } }
I20260621 07:53:26.895555 23860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07376db29096403ba41c0498caea36d5" candidate_uuid: "96c0adedb35847d6984271ab7bf8b64f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f30b1c378e4571a9345eda4f65be54"
I20260621 07:53:26.895673 23860 raft_consensus.cc:3037] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:53:26.896528 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:26.891505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:43193 (local address 127.12.12.3:39621)
0621 07:53:26.891764 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0621 07:53:26.891779 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:53:26.891858 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:53:26.892931 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:53:26.892934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:53:26.892950 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:53:26.893040 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:53:26.893048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.894005 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.894010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:53:26.895171 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:53:26.895179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.896153 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.896156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:53:26.896240 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:26.896262 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:53:26.896329 (+ 67us) client_negotiation.cc:240] Negotiation successful
0621 07:53:26.896395 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260621 07:53:26.896926 24058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:26.891790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:39621 (local address 127.12.12.1:43193)
0621 07:53:26.892683 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0621 07:53:26.892690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:53:26.892706 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:53:26.892756 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:53:26.892760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:53:26.892816 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:53:26.892925 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:53:26.893175 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.893875 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.895315 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.895884 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.895979 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:26.896058 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 07:53:26.896726 (+ 668us) server_negotiation.cc:299] Negotiation successful
0621 07:53:26.896806 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809}
I20260621 07:53:26.897069 23860 raft_consensus.cc:2454] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96c0adedb35847d6984271ab7bf8b64f in term 1.
I20260621 07:53:26.897353 23928 leader_election.cc:302] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96c0adedb35847d6984271ab7bf8b64f, f8f30b1c378e4571a9345eda4f65be54; no voters:
I20260621 07:53:26.897543 24067 leader_election.cc:288] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [CANDIDATE]: Term 1 election: Requested vote from peers f8f30b1c378e4571a9345eda4f65be54 (127.12.12.2:46641), ef5f47e50ed54f6e91d2473a7ec749d3 (127.12.12.1:43193)
I20260621 07:53:26.897529 23723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07376db29096403ba41c0498caea36d5" candidate_uuid: "96c0adedb35847d6984271ab7bf8b64f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5f47e50ed54f6e91d2473a7ec749d3" is_pre_election: true
I20260621 07:53:26.897631 24067 raft_consensus.cc:2781] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:53:26.897706 24067 raft_consensus.cc:686] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [term 1 LEADER]: Becoming Leader. State: Replica: 96c0adedb35847d6984271ab7bf8b64f, State: Running, Role: LEADER
I20260621 07:53:26.897847 23723 raft_consensus.cc:2454] T 07376db29096403ba41c0498caea36d5 P ef5f47e50ed54f6e91d2473a7ec749d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96c0adedb35847d6984271ab7bf8b64f in term 0.
I20260621 07:53:26.897876 23724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07376db29096403ba41c0498caea36d5" candidate_uuid: "96c0adedb35847d6984271ab7bf8b64f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5f47e50ed54f6e91d2473a7ec749d3"
I20260621 07:53:26.897989 23724 raft_consensus.cc:3037] T 07376db29096403ba41c0498caea36d5 P ef5f47e50ed54f6e91d2473a7ec749d3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:53:26.898064 24067 consensus_queue.cc:237] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c0adedb35847d6984271ab7bf8b64f" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 38869 } } peers { permanent_uuid: "f8f30b1c378e4571a9345eda4f65be54" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46641 } } peers { permanent_uuid: "ef5f47e50ed54f6e91d2473a7ec749d3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43193 } }
I20260621 07:53:26.899286 23724 raft_consensus.cc:2454] T 07376db29096403ba41c0498caea36d5 P ef5f47e50ed54f6e91d2473a7ec749d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96c0adedb35847d6984271ab7bf8b64f in term 1.
I20260621 07:53:26.902201 23555 catalog_manager.cc:5495] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f reported cstate change: term changed from 0 to 1, leader changed from <none> to 96c0adedb35847d6984271ab7bf8b64f (127.12.12.3). New cstate: current_term: 1 leader_uuid: "96c0adedb35847d6984271ab7bf8b64f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96c0adedb35847d6984271ab7bf8b64f" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 38869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8f30b1c378e4571a9345eda4f65be54" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5f47e50ed54f6e91d2473a7ec749d3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43193 } health_report { overall_health: UNKNOWN } } }
Jun 21 07:53:26 dist-test-slave-lbpt krb5kdc[23080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028405, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:53:26.934620 24054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:53:26.929140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.12.12.3:38869)
0621 07:53:26.929268 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 07:53:26.929273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:53:26.929286 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:53:26.929441 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:53:26.929444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:53:26.929516 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:53:26.929675 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:53:26.930098 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.931039 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.932393 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:53:26.932728 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:53:26.932823 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:53:26.933259 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:53:26.933830 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:53:26.933958 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 07:53:26.934371 (+ 413us) server_negotiation.cc:299] Negotiation successful
0621 07:53:26.934456 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:53:26.943059 23860 raft_consensus.cc:1261] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54 [term 1 FOLLOWER]: Refusing update from remote peer 96c0adedb35847d6984271ab7bf8b64f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:53:26.943058 23724 raft_consensus.cc:1261] T 07376db29096403ba41c0498caea36d5 P ef5f47e50ed54f6e91d2473a7ec749d3 [term 1 FOLLOWER]: Refusing update from remote peer 96c0adedb35847d6984271ab7bf8b64f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:53:26.943902 24069 consensus_queue.cc:1035] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5f47e50ed54f6e91d2473a7ec749d3" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:53:26.944096 24067 consensus_queue.cc:1035] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f30b1c378e4571a9345eda4f65be54" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 07:53:26.946576 24041 tablet.cc:2369] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:53:26.961133 24077 mvcc.cc:205] Tried to move back new op lower bound from 7299188354828578816 to 7299188354657587200. Current Snapshot: MvccSnapshot[applied={T|T < 7299188354828578816}]
I20260621 07:53:26.961324 24072 mvcc.cc:205] Tried to move back new op lower bound from 7299188354828578816 to 7299188354657587200. Current Snapshot: MvccSnapshot[applied={T|T < 7299188354828578816}]
I20260621 07:53:26.976933 23555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:53:26.977126 23555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:53:26.979792 23555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb6e11d7fe394d86a57ea7f47292b5c1: Sending DeleteTablet for 3 replicas of tablet 07376db29096403ba41c0498caea36d5
I20260621 07:53:26.981004 23700 tablet_service.cc:1514] Processing DeleteTablet for tablet 07376db29096403ba41c0498caea36d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:53:26 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:50934
I20260621 07:53:26.981037 23823 tablet_service.cc:1514] Processing DeleteTablet for tablet 07376db29096403ba41c0498caea36d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:53:26 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:43808
I20260621 07:53:26.981887 24084 tablet_replica.cc:330] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54: stopping tablet replica
I20260621 07:53:26.981956 23973 tablet_service.cc:1514] Processing DeleteTablet for tablet 07376db29096403ba41c0498caea36d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:53:26 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:41290
I20260621 07:53:26.982228 24084 raft_consensus.cc:2229] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:53:26.982542 24084 raft_consensus.cc:2258] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:53:26.982913 24083 tablet_replica.cc:330] T 07376db29096403ba41c0498caea36d5 P ef5f47e50ed54f6e91d2473a7ec749d3: stopping tablet replica
I20260621 07:53:26.982975 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 23637
I20260621 07:53:26.983214 24083 raft_consensus.cc:2229] T 07376db29096403ba41c0498caea36d5 P ef5f47e50ed54f6e91d2473a7ec749d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:53:26.983836 24085 tablet_replica.cc:330] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f: stopping tablet replica
I20260621 07:53:26.984103 24085 raft_consensus.cc:2229] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:53:26.984573 24085 raft_consensus.cc:2258] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:53:26.986173 24085 ts_tablet_manager.cc:1905] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:53:26.989987 24085 ts_tablet_manager.cc:1918] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:53:26.990103 24085 log.cc:1196] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/07376db29096403ba41c0498caea36d5
I20260621 07:53:26.990497 24085 ts_tablet_manager.cc:1939] T 07376db29096403ba41c0498caea36d5 P 96c0adedb35847d6984271ab7bf8b64f: Deleting consensus metadata
I20260621 07:53:26.991878 23543 catalog_manager.cc:4841] TS 96c0adedb35847d6984271ab7bf8b64f (127.12.12.3:38869): tablet 07376db29096403ba41c0498caea36d5 (table test-table [id=fce9b4b9f9804d9b9df9c700f2df1a36]) successfully deleted
I20260621 07:53:26.993670 24084 ts_tablet_manager.cc:1905] T 07376db29096403ba41c0498caea36d5 P f8f30b1c378e4571a9345eda4f65be54: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260621 07:53:26.997948 23541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.1:43193 (error 108)
W20260621 07:53:26.999356 24057 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:53:26.998842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:43193 (local address 127.0.0.1:50942)
0621 07:53:26.999005 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 07:53:26.999023 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:53:26.999123 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:53:26.999225 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:43193: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 07:53:26.999574 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 23776
W20260621 07:53:26.999717 23541 catalog_manager.cc:4584] TS ef5f47e50ed54f6e91d2473a7ec749d3 (127.12.12.1:43193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07376db29096403ba41c0498caea36d5: Network error: Client connection negotiation failed: client connection to 127.12.12.1:43193: BlockingWrite error: write error: Connection reset by peer (error 104)
W20260621 07:53:27.008879 24057 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:53:27.008261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:46641 (local address 127.0.0.1:43818)
0621 07:53:27.008449 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 07:53:27.008465 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:53:27.008559 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:53:27.008765 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.2:46641: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":110}
I20260621 07:53:27.009670 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 23911
I20260621 07:53:27.017864 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 23524
I20260621 07:53:27.027480 12336 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 07:53:27.373052 12336 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 07:53:27.373183 12336 mini_ranger.cc:66] Stopping Ranger...
I20260621 07:53:27.718562 12336 mini_ranger.cc:68] Stopped Ranger
2026-06-21 07:53:27.718 UTC [23110] LOG: received smart shutdown request
2026-06-21 07:53:27.722 UTC [23110] LOG: background worker "logical replication launcher" (PID 23119) exited with exit code 1
2026-06-21 07:53:27.722 UTC [23114] LOG: shutting down
2026-06-21 07:53:27.890 UTC [23110] LOG: database system is shut down
2026-06-21T07:53:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45806 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:53:28 dist-test-slave-lbpt krb5kdc[24095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:53:28 dist-test-slave-lbpt krb5kdc[24095](info): set up 2 sockets
Jun 21 07:53:28 dist-test-slave-lbpt krb5kdc[24095](info): commencing operation
krb5kdc: starting...
W20260621 07:53:29.062606 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:53:29 dist-test-slave-lbpt krb5kdc[24095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:53:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:53:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.12.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.12.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.12.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/rangeradmin_127.12.12.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.12.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/rangeradmin_127.12.12.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.12.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.12.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.12.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/rangerlookup_127.12.12.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.12.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/rangerlookup_127.12.12.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.12.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/HTTP_127.12.12.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.12.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/HTTP_127.12.12.20@KRBTEST.COM.keytab.
I20260621 07:53:29.238924 12336 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskzeuNYm/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/postgres -l logfile start
2026-06-21 07:53:31.821 UTC [24125] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 07:53:31.822 UTC [24125] LOG: listening on IPv4 address "127.12.12.20", port 52251
2026-06-21 07:53:31.825 UTC [24125] LOG: listening on Unix socket "/tmp/.s.PGSQL.52251"
2026-06-21 07:53:31.861 UTC [24128] LOG: database system was shut down at 2026-06-21 07:53:30 UTC
2026-06-21 07:53:31.865 UTC [24125] LOG: database system is ready to accept connections
I20260621 07:53:32.814371 12336 mini_postgres.cc:96] Postgres bound to 52251
2026-06-21 07:53:32.820 UTC [24136] FATAL: database "slave" does not exist
127.12.12.20:52251 - accepting connections
I20260621 07:53:32.821178 12336 mini_ranger.cc:161] Starting Ranger...
I20260621 07:53:32.836243 12336 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 07:53:33.298504 12336 mini_ranger.cc:87] Created ranger Postgres database
I20260621 07:53:33.298588 12336 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-admin
2026-06-21 07:53:33,577 [I] DB FLAVOR :POSTGRES
2026-06-21 07:53:33,578 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 07:53:33,578 [I] Checking connection..
2026-06-21 07:53:33,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 07:53:33,830 [I] Checking connection passed.
2026-06-21 07:53:33,830 [I] --------- Verifying version history table ---------
2026-06-21 07:53:33,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:53:34,080 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 07:53:34,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:53:34,329 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 07:53:34,330 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 07:53:34,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 07:53:34.578 UTC [24200] WARNING: there is no transaction in progress
2026-06-21 07:53:34,585 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 07:53:34,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:53:34,824 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 07:53:34,824 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 07:53:34,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 07:53:35,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 07:53:35,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lbpt.c.gcp-upstream.internal','N') ;"
2026-06-21 07:53:35,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 07:53:35,852 [I] Table x_portal_user does not exist in database ranger
2026-06-21 07:53:35,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 07:53:36,105 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 07:53:36,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 07:53:36,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 07:53:36,610 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 07:53:36,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 07:53:37.602 UTC [24308] WARNING: there is no transaction in progress
2026-06-21 07:53:37.622 UTC [24308] WARNING: there is no transaction in progress
2026-06-21 07:53:37.641 UTC [24308] WARNING: there is no transaction in progress
2026-06-21 07:53:37.659 UTC [24308] WARNING: there is no transaction in progress
2026-06-21 07:53:37.677 UTC [24308] WARNING: there is no transaction in progress
2026-06-21 07:53:37.696 UTC [24308] WARNING: there is no transaction in progress
2026-06-21 07:53:37.755 UTC [24308] WARNING: there is no transaction in progress
2026-06-21 07:53:37.762 UTC [24308] WARNING: there is no transaction in progress
2026-06-21 07:53:37.778 UTC [24308] WARNING: there is no transaction in progress
2026-06-21 07:53:37.790 UTC [24308] WARNING: there is no transaction in progress
2026-06-21 07:53:37.802 UTC [24308] WARNING: there is no transaction in progress
2026-06-21 07:53:37.817 UTC [24308] WARNING: there is no transaction in progress
2026-06-21 07:53:37.828 UTC [24308] WARNING: there is no transaction in progress
2026-06-21 07:53:37.836 UTC [24308] WARNING: there is no transaction in progress
2026-06-21 07:53:37.844 UTC [24308] WARNING: there is no transaction in progress
2026-06-21 07:53:37.851 UTC [24308] WARNING: there is no transaction in progress
2026-06-21 07:53:38,515 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 07:53:38,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 07:53:38,750 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 07:53:38,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 07:53:38,996 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 07:53:38,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 07:53:39,255 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 07:53:39,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 07:53:39,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 07:53:39,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lbpt.c.gcp-upstream.internal';"
2026-06-21 07:53:40,068 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 07:53:40,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 07:53:40,316 [I] DB_PATCHES have already been applied
I20260621 07:53:40.321792 12336 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-admin:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskzeuNYm/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 07:53:40.321859 12336 mini_ranger.cc:193] Using host: 127.12.12.20
I20260621 07:53:40.325371 12336 mini_ranger.cc:227] Ranger admin URL: http://127.12.12.20:51713
Jun 21, 2026 7:53:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 7:53:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 7:53:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 7:53:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 7:53:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 7:53:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 7:53:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 07:53:41.431339 12336 mini_ranger.cc:160] Time spent starting Ranger: real 8.610s user 0.001s sys 0.009s
Jun 21, 2026 7:54:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 7:54:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 21, 2026 7:54:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 7:54:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 7:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 7:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 7:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 7:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 7:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 7:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 7:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 7:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 7:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 7:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 7:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 7:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 7:54:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 7:54:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 07:54:03.234642 12336 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.12.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.12.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.12.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/rangerkms_127.12.12.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.12.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/rangerkms_127.12.12.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.12.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.12.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/HTTP_127.12.12.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.12.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/HTTP_127.12.12.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 07:54:03 dist-test-slave-lbpt krb5kdc[24095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028443, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 07:54:03.338224 12336 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 07:54:03.349128 12336 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 07:54:04.224164 12336 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 07:54:04.224251 12336 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-kms
I20260621 07:54:04.224272 12336 mini_ranger_kms.cc:227] Using postgres at 127.12.12.20:52251
2026-06-21 07:54:04,392 [I] DB FLAVOR :POSTGRES
2026-06-21 07:54:04,392 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 07:54:04,392 [I] Checking connection
2026-06-21 07:54:04,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 07:54:04,629 [I] connection success
2026-06-21 07:54:04,630 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 07:54:04,630 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 07:54:04,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 07:54:04,866 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 07:54:04,866 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 07:54:04,867 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 07:54:04,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.20:52251/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 07:54:05,138 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 07:54:05.593454 12336 mini_ranger_kms.cc:310] Created kms service
I20260621 07:54:05.778561 12336 mini_ranger_kms.cc:326] Created kudu user
I20260621 07:54:05.826690 12336 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 07:54:05 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028443, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:54:06.242996 12336 mini_ranger_kms.cc:384] Added ranger policy
I20260621 07:54:06.243261 12336 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-kms:/tmp/dist-test-taskzeuNYm/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskzeuNYm/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 07:54:06.243291 12336 mini_ranger_kms.cc:240] Using host: 127.12.12.20
I20260621 07:54:06.247682 12336 mini_ranger_kms.cc:276] Ranger KMS PID: 24488
I20260621 07:54:06.247794 12336 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.12.20:58065
Jun 21, 2026 7:54:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 7:54:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 7:54:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 7:54:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 7:54:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 7:54:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260621 07:54:08.546975 12336 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.209s user 0.005s sys 0.146s
I20260621 07:54:08.547116 12336 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 07:54:08.547192 12336 mini_ranger_kms.cc:401] 127.12.12.20:58065/kms/v1/keys
Jun 21 07:54:08 dist-test-slave-lbpt krb5kdc[24095](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782028448, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.12.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 21 07:54:09 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028448, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.12.20@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
Jun 21 07:54:09 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028448, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.12.20@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
Jun 21 07:54:11 dist-test-slave-lbpt krb5kdc[24095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:54:11.787194 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:36199
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:36199
--ranger_config_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.20:58065/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:54:11.901052 24539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:11.901329 24539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:11.901367 24539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:11.905522 24539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:54:11.905601 24539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:11.905622 24539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:11.905637 24539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:54:11.905653 24539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:54:11.911417 24539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.12.20:58065/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:36199
--ranger_config_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:36199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:11.912621 24539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:11.913757 24539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:11.922581 24547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:11.922600 24545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:11.922562 24544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:11.923158 24539 server_base.cc:1034] running on GCE node
I20260621 07:54:11.923822 24539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:11.925272 24539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:11.926481 24539 hybrid_clock.cc:648] HybridClock initialized: now 1782028451926447 us; error 45 us; skew 500 ppm
Jun 21 07:54:11 dist-test-slave-lbpt krb5kdc[24095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028451, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:11.929813 24539 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:54:11.930936 24539 webserver.cc:460] Webserver started at http://127.12.12.62:45765/ using document root <none> and password file <none>
I20260621 07:54:11.931488 24539 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:11.931574 24539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:11.931782 24539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:54:11 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028451, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:54:11.996235 24539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "4e10998b06b445138307dd67d8a406b8"
format_stamp: "Formatted at 2026-06-21 07:54:11 on dist-test-slave-lbpt"
server_key: "c1c2de51cbaac444bbb9d1ed2188fd5b"
server_key_iv: "45a11eecae7fdfac72d705fac79f1756"
server_key_version: "kuduclusterkey@0"
I20260621 07:54:11.997900 24539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "4e10998b06b445138307dd67d8a406b8"
format_stamp: "Formatted at 2026-06-21 07:54:11 on dist-test-slave-lbpt"
server_key: "c1c2de51cbaac444bbb9d1ed2188fd5b"
server_key_iv: "45a11eecae7fdfac72d705fac79f1756"
server_key_version: "kuduclusterkey@0"
I20260621 07:54:12.003525 24539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:54:12.021375 24554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:12.022845 24539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:54:12.022985 24539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "4e10998b06b445138307dd67d8a406b8"
format_stamp: "Formatted at 2026-06-21 07:54:11 on dist-test-slave-lbpt"
server_key: "c1c2de51cbaac444bbb9d1ed2188fd5b"
server_key_iv: "45a11eecae7fdfac72d705fac79f1756"
server_key_version: "kuduclusterkey@0"
I20260621 07:54:12.023118 24539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:12.045943 24539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:12.049479 24539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:12.049719 24539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:12.059371 24539 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:36199
I20260621 07:54:12.059374 24606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:36199 every 8 connection(s)
I20260621 07:54:12.060501 24539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:54:12.063746 24607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:12.065282 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 24539
I20260621 07:54:12.065408 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:54:12.065733 12336 external_mini_cluster.cc:1428] Setting key ebe8f47be180ee6e9193fbc70ba2d771
I20260621 07:54:12.072921 24607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8: Bootstrap starting.
I20260621 07:54:12.075809 24607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8: Neither blocks nor log segments found. Creating new log.
Jun 21 07:54:12 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:12.076846 24607 log.cc:824] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:12.079545 24607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8: No bootstrap required, opened a new log
I20260621 07:54:12.083988 24610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:12.067274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.12.12.62:36199)
0621 07:54:12.067701 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:12.067722 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:12.067768 (+ 46us) server_negotiation.cc:407] Connection header received
0621 07:54:12.068629 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:12.068657 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:12.069138 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:12.069554 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:12.070485 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:12.072134 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:12.073351 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:12.074083 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:12.074271 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:12.077537 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:12.077543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:12.077558 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:12.077589 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:12.080524 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:12.081063 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:12.081070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:12.081084 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:12.081235 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:12.081554 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:12.081557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:12.081560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:12.082102 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:12.082302 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:12.082637 (+ 335us) server_negotiation.cc:299] Negotiation successful
0621 07:54:12.082957 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":146,"threads_started":1}
I20260621 07:54:12.084208 24607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e10998b06b445138307dd67d8a406b8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36199 } }
I20260621 07:54:12.084522 24607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:12.084617 24607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e10998b06b445138307dd67d8a406b8, State: Initialized, Role: FOLLOWER
I20260621 07:54:12.085148 24607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e10998b06b445138307dd67d8a406b8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36199 } }
I20260621 07:54:12.085337 24607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:54:12.085424 24607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:54:12.085534 24607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:12.086695 24607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e10998b06b445138307dd67d8a406b8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36199 } }
I20260621 07:54:12.087061 24607 leader_election.cc:302] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e10998b06b445138307dd67d8a406b8; no voters:
I20260621 07:54:12.087430 24607 leader_election.cc:288] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:54:12.087510 24612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:54:12.087751 24612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 1 LEADER]: Becoming Leader. State: Replica: 4e10998b06b445138307dd67d8a406b8, State: Running, Role: LEADER
I20260621 07:54:12.088138 24612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e10998b06b445138307dd67d8a406b8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36199 } }
I20260621 07:54:12.088820 24607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:54:12.089958 24613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e10998b06b445138307dd67d8a406b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e10998b06b445138307dd67d8a406b8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36199 } } }
I20260621 07:54:12.090097 24613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [sys.catalog]: This master's current role is: LEADER
I20260621 07:54:12.090322 24613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e10998b06b445138307dd67d8a406b8. Latest consensus state: current_term: 1 leader_uuid: "4e10998b06b445138307dd67d8a406b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e10998b06b445138307dd67d8a406b8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36199 } } }
I20260621 07:54:12.090400 24613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [sys.catalog]: This master's current role is: LEADER
I20260621 07:54:12.091361 24607 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 07:54:12.091863 24621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:54:12.098740 24621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:54:12.106838 24621 catalog_manager.cc:1347] Generated new cluster ID: 59b25af0c4874a7ebc1af8a15fbbfcfe
I20260621 07:54:12.107015 24621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:54:12.147336 24621 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:54:12.149055 24621 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:54:12.158226 24621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8: Generated new TSK 0
I20260621 07:54:12.159279 24621 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:54:12.837770 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.20:58065/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:36199
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:54:12.957937 24652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:12.958248 24652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:12.958335 24652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:12.962854 24652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:12.962940 24652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:12.963033 24652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:54:12.969439 24652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.12.20:58065/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:36199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:12.970664 24652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:12.971714 24652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:12.981599 24658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:12.981678 24660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:12.981603 24657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:12.982172 24652 server_base.cc:1034] running on GCE node
I20260621 07:54:12.982623 24652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:12.983311 24652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:12.984488 24652 hybrid_clock.cc:648] HybridClock initialized: now 1782028452984456 us; error 47 us; skew 500 ppm
Jun 21 07:54:12 dist-test-slave-lbpt krb5kdc[24095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028452, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:12.987625 24652 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:54:12.988921 24652 webserver.cc:460] Webserver started at http://127.12.12.1:33331/ using document root <none> and password file <none>
I20260621 07:54:12.989758 24652 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:12.989899 24652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:12.990197 24652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:54:12 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028452, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:54:13.014391 24652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "10e714f3b7174620bdfa444bd4f817bc"
format_stamp: "Formatted at 2026-06-21 07:54:13 on dist-test-slave-lbpt"
server_key: "31d624f9bf939eb2e02603c7642bde17"
server_key_iv: "c78b2cf35b548bbb84dc6e80589b9d27"
server_key_version: "kuduclusterkey@0"
I20260621 07:54:13.015470 24652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "10e714f3b7174620bdfa444bd4f817bc"
format_stamp: "Formatted at 2026-06-21 07:54:13 on dist-test-slave-lbpt"
server_key: "31d624f9bf939eb2e02603c7642bde17"
server_key_iv: "c78b2cf35b548bbb84dc6e80589b9d27"
server_key_version: "kuduclusterkey@0"
I20260621 07:54:13.020037 24652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:54:13.044725 24669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:13.046916 24652 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260621 07:54:13.047154 24652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "10e714f3b7174620bdfa444bd4f817bc"
format_stamp: "Formatted at 2026-06-21 07:54:13 on dist-test-slave-lbpt"
server_key: "31d624f9bf939eb2e02603c7642bde17"
server_key_iv: "c78b2cf35b548bbb84dc6e80589b9d27"
server_key_version: "kuduclusterkey@0"
I20260621 07:54:13.047394 24652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:13.076225 24652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:13.079710 24652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:13.079988 24652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:13.080760 24652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:13.081862 24652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:54:13.081979 24652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:13.082111 24652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:54:13.082197 24652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:13.096102 24652 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:38717
I20260621 07:54:13.097353 24652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:54:13.102203 24782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:38717 every 8 connection(s)
I20260621 07:54:13.106196 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 24652
I20260621 07:54:13.106329 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:54:13.106662 12336 external_mini_cluster.cc:1428] Setting key 1bfc0ed395b9b498ca0c29ed4e01f43d
Jun 21 07:54:13 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028452, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
I20260621 07:54:13.121733 24785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:13.104554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:40087 (local address 127.12.12.62:36199)
0621 07:54:13.105900 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:13.105909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:13.105930 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:54:13.107878 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:13.107883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:13.107939 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:13.108029 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:13.111140 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:13.112029 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:13.113382 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:13.113805 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:13.113945 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:13.116928 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:13.116931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:13.116933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:13.116960 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:13.119026 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:13.119759 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:13.119764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:13.119770 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:13.119852 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:13.120267 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:13.120271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:13.120273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:13.120518 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:13.120618 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:13.121391 (+ 773us) server_negotiation.cc:299] Negotiation successful
0621 07:54:13.121544 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1251,"thread_start_us":151,"threads_started":1}
I20260621 07:54:13.123920 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:13.104767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:36199 (local address 127.12.12.1:40087)
0621 07:54:13.105340 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:13.105404 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:13.107609 (+ 2205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:13.109650 (+ 2041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:13.109659 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:13.110030 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:13.110912 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:13.110937 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:13.112210 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:13.112214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:13.113230 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:13.113241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:13.114217 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:13.114222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:13.114505 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:13.115236 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:13.115254 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:13.116722 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:13.119237 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:13.119245 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:13.119270 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:13.119601 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:13.119984 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:13.119988 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:13.119989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:13.120144 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:13.120676 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:13.120684 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:13.121063 (+ 379us) client_negotiation.cc:769] Sending connection context
0621 07:54:13.122315 (+ 1252us) client_negotiation.cc:240] Negotiation successful
0621 07:54:13.122749 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":141,"threads_started":1}
I20260621 07:54:13.125914 24783 heartbeater.cc:347] Connected to a master server at 127.12.12.62:36199
I20260621 07:54:13.126190 24783 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:13.126873 24783 heartbeater.cc:511] Master 127.12.12.62:36199 requested a full tablet report, sending...
I20260621 07:54:13.128829 24570 ts_manager.cc:194] Registered new tserver with Master: 10e714f3b7174620bdfa444bd4f817bc (127.12.12.1:38717)
I20260621 07:54:13.130498 24570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:40087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:54:13.153273 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.20:58065/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:36199
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:54:13 dist-test-slave-lbpt krb5kdc[24095](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782028453, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260621 07:54:13.318118 24791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:13.318580 24791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:13.318655 24791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:13.324950 24791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:13.325044 24791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:13.325132 24791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:54:13.333326 24791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.12.20:58065/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:36199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:13.334748 24791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:13.336182 24791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:13.348760 24798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:13.348760 24797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:13.348974 24800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:13.350701 24791 server_base.cc:1034] running on GCE node
I20260621 07:54:13.351365 24791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:13.352263 24791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:13.353474 24791 hybrid_clock.cc:648] HybridClock initialized: now 1782028453353449 us; error 41 us; skew 500 ppm
Jun 21 07:54:13 dist-test-slave-lbpt krb5kdc[24095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028453, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:13.357357 24791 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:54:13.358527 24791 webserver.cc:460] Webserver started at http://127.12.12.2:46581/ using document root <none> and password file <none>
I20260621 07:54:13.359251 24791 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:13.359340 24791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:13.359570 24791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:54:13 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028453, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:54:13.380087 24791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "ef0e9ecb18d44a7880988be012abe1fe"
format_stamp: "Formatted at 2026-06-21 07:54:13 on dist-test-slave-lbpt"
server_key: "4a63645bf99dc9875c7b3bfa3c607f5d"
server_key_iv: "c587c15d2f936f95dfdc316fd3286b84"
server_key_version: "kuduclusterkey@0"
I20260621 07:54:13.380937 24791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "ef0e9ecb18d44a7880988be012abe1fe"
format_stamp: "Formatted at 2026-06-21 07:54:13 on dist-test-slave-lbpt"
server_key: "4a63645bf99dc9875c7b3bfa3c607f5d"
server_key_iv: "c587c15d2f936f95dfdc316fd3286b84"
server_key_version: "kuduclusterkey@0"
I20260621 07:54:13.388208 24791 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260621 07:54:13.406937 24807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:13.408869 24791 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260621 07:54:13.409089 24791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "ef0e9ecb18d44a7880988be012abe1fe"
format_stamp: "Formatted at 2026-06-21 07:54:13 on dist-test-slave-lbpt"
server_key: "4a63645bf99dc9875c7b3bfa3c607f5d"
server_key_iv: "c587c15d2f936f95dfdc316fd3286b84"
server_key_version: "kuduclusterkey@0"
I20260621 07:54:13.409332 24791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:13.443262 24791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:13.447417 24791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:13.447703 24791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:13.448513 24791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:13.449635 24791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:54:13.449746 24791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:13.449853 24791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:54:13.449924 24791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:13.461380 24791 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:44905
I20260621 07:54:13.461453 24920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:44905 every 8 connection(s)
I20260621 07:54:13.462740 24791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:54:13.471659 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 24791
I20260621 07:54:13.471805 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:54:13.472144 12336 external_mini_cluster.cc:1428] Setting key 60494e71d3b7e3ad765111d0164a5577
Jun 21 07:54:13 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028453, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:13.481649 24785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:13.466837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:36499 (local address 127.12.12.62:36199)
0621 07:54:13.467004 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:13.467009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:13.467029 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:54:13.467741 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:13.467744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:13.467822 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:13.467922 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:13.469489 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:13.470296 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:13.472005 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:13.472414 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:13.472521 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:13.476377 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:13.476384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:13.476388 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:13.476425 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:13.478540 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:13.479331 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:13.479336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:13.479339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:13.479450 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:13.479994 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:13.479998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:13.480001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:13.480291 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:13.480388 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:13.481325 (+ 937us) server_negotiation.cc:299] Negotiation successful
0621 07:54:13.481474 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:54:13.486658 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:13.465597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:36199 (local address 127.12.12.2:36499)
0621 07:54:13.466222 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:13.466290 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:13.467457 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:13.468161 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:13.468172 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:13.468559 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:13.469288 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:13.469309 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:13.470492 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:13.470497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:13.471852 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:13.471863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:13.472785 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:13.472790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:13.473035 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:13.474024 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:13.474054 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:13.476119 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:13.478699 (+ 2580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:13.478709 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:13.478734 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:13.479188 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:13.479607 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:13.479612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:13.479615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:13.479863 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:13.480470 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:13.480478 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:13.480938 (+ 460us) client_negotiation.cc:769] Sending connection context
0621 07:54:13.481210 (+ 272us) client_negotiation.cc:240] Negotiation successful
0621 07:54:13.481580 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":166,"threads_started":1}
I20260621 07:54:13.491834 24921 heartbeater.cc:347] Connected to a master server at 127.12.12.62:36199
I20260621 07:54:13.492172 24921 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:13.492807 24921 heartbeater.cc:511] Master 127.12.12.62:36199 requested a full tablet report, sending...
I20260621 07:54:13.494122 24570 ts_manager.cc:194] Registered new tserver with Master: ef0e9ecb18d44a7880988be012abe1fe (127.12.12.2:44905)
I20260621 07:54:13.494925 24570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:36499
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:54:13.536955 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.20:58065/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:36199
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:54:13.708797 24926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:13.709671 24926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:13.709811 24926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:13.715735 24926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:13.715896 24926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:13.716058 24926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:54:13.724152 24926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.12.20:58065/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:36199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:13.725835 24926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:13.727216 24926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:54:13.740569 24926 server_base.cc:1034] running on GCE node
W20260621 07:54:13.740481 24931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:13.741086 24932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:13.741384 24934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:13.741890 24926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:13.742703 24926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:13.750108 24926 hybrid_clock.cc:648] HybridClock initialized: now 1782028453750026 us; error 87 us; skew 500 ppm
Jun 21 07:54:13 dist-test-slave-lbpt krb5kdc[24095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028453, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 21 07:54:13 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028453, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:54:13.754123 24926 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:54:13.755445 24926 webserver.cc:460] Webserver started at http://127.12.12.3:38925/ using document root <none> and password file <none>
I20260621 07:54:13.756312 24926 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:13.756500 24926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:13.757154 24926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:54:13 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028453, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:54:13.787813 24926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "dbb2c727ab634a08942c32a89ff60e54"
format_stamp: "Formatted at 2026-06-21 07:54:13 on dist-test-slave-lbpt"
server_key: "3a8dfd96d2f5c1f4c1135024a7ddc736"
server_key_iv: "8144616b5a3dc538b1fbff079ea6e5c7"
server_key_version: "kuduclusterkey@0"
I20260621 07:54:13.788661 24926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "dbb2c727ab634a08942c32a89ff60e54"
format_stamp: "Formatted at 2026-06-21 07:54:13 on dist-test-slave-lbpt"
server_key: "3a8dfd96d2f5c1f4c1135024a7ddc736"
server_key_iv: "8144616b5a3dc538b1fbff079ea6e5c7"
server_key_version: "kuduclusterkey@0"
I20260621 07:54:13.804155 24926 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.006s sys 0.000s
I20260621 07:54:13.834877 24941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:13.836788 24926 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.001s
I20260621 07:54:13.836937 24926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "dbb2c727ab634a08942c32a89ff60e54"
format_stamp: "Formatted at 2026-06-21 07:54:13 on dist-test-slave-lbpt"
server_key: "3a8dfd96d2f5c1f4c1135024a7ddc736"
server_key_iv: "8144616b5a3dc538b1fbff079ea6e5c7"
server_key_version: "kuduclusterkey@0"
I20260621 07:54:13.837085 24926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:13.879087 24926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:13.883196 24926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:13.883430 24926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:13.884146 24926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:13.885156 24926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:54:13.885228 24926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:13.885329 24926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:54:13.885375 24926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:13.897502 24926 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:42347
I20260621 07:54:13.897522 25054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:42347 every 8 connection(s)
I20260621 07:54:13.898777 24926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:54:13.902693 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 24926
I20260621 07:54:13.902848 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:54:13.903163 12336 external_mini_cluster.cc:1428] Setting key 10a7d7bcf8dfebdeeb397a0e8df7ed1c
Jun 21 07:54:13 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028453, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:13.922175 24785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:13.900946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:57381 (local address 127.12.12.62:36199)
0621 07:54:13.901111 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:13.901114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:13.902186 (+ 1072us) server_negotiation.cc:407] Connection header received
0621 07:54:13.903774 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:13.903778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:13.903839 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:13.903901 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:13.905602 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:13.906137 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:13.911956 (+ 5819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:13.912346 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:13.912434 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:13.917657 (+ 5223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:13.917661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:13.917664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:13.917694 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:13.919554 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:13.920228 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:13.920232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:13.920234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:13.920319 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:13.920742 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:13.920745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:13.920747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:13.920980 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:13.921077 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:13.921852 (+ 775us) server_negotiation.cc:299] Negotiation successful
0621 07:54:13.921992 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 07:54:13.924264 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:13.901460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:36199 (local address 127.12.12.3:57381)
0621 07:54:13.902014 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:13.902062 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:13.903281 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:13.904346 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:13.904357 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:13.904749 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:13.905460 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:13.905478 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:13.906263 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:13.906267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:13.911769 (+ 5502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:13.911786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:13.912657 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:13.912662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:13.912878 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:13.913777 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:13.913798 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:13.917448 (+ 3650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:13.919698 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:13.919706 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:13.919727 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:13.920110 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:13.920425 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:13.920428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:13.920430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:13.920632 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:13.921148 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:13.921156 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:13.921528 (+ 372us) client_negotiation.cc:769] Sending connection context
0621 07:54:13.922720 (+ 1192us) client_negotiation.cc:240] Negotiation successful
0621 07:54:13.923108 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":139,"threads_started":1}
I20260621 07:54:13.925918 25055 heartbeater.cc:347] Connected to a master server at 127.12.12.62:36199
I20260621 07:54:13.926206 25055 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:13.926860 25055 heartbeater.cc:511] Master 127.12.12.62:36199 requested a full tablet report, sending...
I20260621 07:54:13.929803 24570 ts_manager.cc:194] Registered new tserver with Master: dbb2c727ab634a08942c32a89ff60e54 (127.12.12.3:42347)
I20260621 07:54:13.930737 24570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:57381
I20260621 07:54:13.941308 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:54:13.942291 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 24652
I20260621 07:54:13.951555 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 24791
I20260621 07:54:13.957566 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 24926
I20260621 07:54:13.962499 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 24539
I20260621 07:54:13.968029 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:36199
--webserver_interface=127.12.12.62
--webserver_port=45765
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:36199
--ranger_config_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.20:58065/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.12.20:58065/kms with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:54:14.086054 25058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:14.086349 25058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:14.086390 25058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:14.090610 25058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:54:14.090689 25058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:14.090709 25058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:14.090727 25058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:54:14.090742 25058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:54:14.096668 25058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.12.20:58065/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:36199
--ranger_config_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:36199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=45765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:14.097872 25058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:14.098975 25058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:14.107579 25064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:14.107739 25066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:14.107764 25063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:14.107831 25058 server_base.cc:1034] running on GCE node
I20260621 07:54:14.108716 25058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:14.109854 25058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:14.111099 25058 hybrid_clock.cc:648] HybridClock initialized: now 1782028454111082 us; error 37 us; skew 500 ppm
Jun 21 07:54:14 dist-test-slave-lbpt krb5kdc[24095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028454, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:14.114102 25058 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:54:14.115127 25058 webserver.cc:460] Webserver started at http://127.12.12.62:45765/ using document root <none> and password file <none>
I20260621 07:54:14.115757 25058 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:14.115861 25058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 07:54:14 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028454, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:54:14.164929 25058 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:54:14.167816 25074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:14.169463 25058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:54:14.169623 25058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "4e10998b06b445138307dd67d8a406b8"
format_stamp: "Formatted at 2026-06-21 07:54:11 on dist-test-slave-lbpt"
server_key: "c1c2de51cbaac444bbb9d1ed2188fd5b"
server_key_iv: "45a11eecae7fdfac72d705fac79f1756"
server_key_version: "kuduclusterkey@0"
I20260621 07:54:14.170373 25058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:14.234710 25058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:14.238196 25058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:14.238376 25058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:14.245895 25058 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:36199
I20260621 07:54:14.245916 25126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:36199 every 8 connection(s)
I20260621 07:54:14.247243 25058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:54:14.251573 25127 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:54:14.253173 25127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8: Bootstrap starting.
I20260621 07:54:14.256824 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 25058
I20260621 07:54:14.257328 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.20:58065/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:38717
--local_ip_for_outbound_sockets=127.12.12.1
--tserver_master_addrs=127.12.12.62:36199
--webserver_port=33331
--webserver_interface=127.12.12.1
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.12.20:58065/kms with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:54:14.270742 25127 log.cc:824] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:14.278083 25127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:54:14.278841 25127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8: Bootstrap complete.
I20260621 07:54:14.283601 25127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e10998b06b445138307dd67d8a406b8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36199 } }
I20260621 07:54:14.284017 25127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e10998b06b445138307dd67d8a406b8, State: Initialized, Role: FOLLOWER
I20260621 07:54:14.284592 25127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e10998b06b445138307dd67d8a406b8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36199 } }
I20260621 07:54:14.284726 25127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:54:14.284821 25127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:54:14.284919 25127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:54:14.287045 25127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e10998b06b445138307dd67d8a406b8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36199 } }
I20260621 07:54:14.287412 25127 leader_election.cc:302] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e10998b06b445138307dd67d8a406b8; no voters:
I20260621 07:54:14.287775 25127 leader_election.cc:288] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 07:54:14.287873 25131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 07:54:14.288129 25131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [term 2 LEADER]: Becoming Leader. State: Replica: 4e10998b06b445138307dd67d8a406b8, State: Running, Role: LEADER
I20260621 07:54:14.288496 25131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e10998b06b445138307dd67d8a406b8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36199 } }
I20260621 07:54:14.289093 25127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:54:14.290045 25133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e10998b06b445138307dd67d8a406b8. Latest consensus state: current_term: 2 leader_uuid: "4e10998b06b445138307dd67d8a406b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e10998b06b445138307dd67d8a406b8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36199 } } }
I20260621 07:54:14.290205 25133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [sys.catalog]: This master's current role is: LEADER
I20260621 07:54:14.290525 25132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4e10998b06b445138307dd67d8a406b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e10998b06b445138307dd67d8a406b8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 36199 } } }
I20260621 07:54:14.290611 25140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:54:14.290618 25132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8 [sys.catalog]: This master's current role is: LEADER
I20260621 07:54:14.294054 25140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:54:14.294647 25127 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 07:54:14.294842 25140 catalog_manager.cc:1259] Loaded cluster ID: 59b25af0c4874a7ebc1af8a15fbbfcfe
I20260621 07:54:14.294878 25140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:54:14.296711 25140 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:54:14.297291 25140 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8: Loaded TSK: 0
I20260621 07:54:14.298249 25140 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:54:14.406059 25129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:14.406559 25129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:14.406658 25129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:14.412703 25129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:14.412833 25129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:14.412953 25129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:54:14.420835 25129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.12.20:58065/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:38717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=33331
--tserver_master_addrs=127.12.12.62:36199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:14.422348 25129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:14.423724 25129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:14.437184 25165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:14.437290 25167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:14.437189 25164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:14.438759 25129 server_base.cc:1034] running on GCE node
I20260621 07:54:14.439248 25129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:14.439950 25129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:14.443253 25129 hybrid_clock.cc:648] HybridClock initialized: now 1782028454443223 us; error 74 us; skew 500 ppm
Jun 21 07:54:14 dist-test-slave-lbpt krb5kdc[24095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028454, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:14.446305 25129 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:54:14.447613 25129 webserver.cc:460] Webserver started at http://127.12.12.1:33331/ using document root <none> and password file <none>
I20260621 07:54:14.448235 25129 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:14.448346 25129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 07:54:14 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028454, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:54:14.474495 25129 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.001s
I20260621 07:54:14.477643 25175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:14.479254 25129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 07:54:14.479368 25129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "10e714f3b7174620bdfa444bd4f817bc"
format_stamp: "Formatted at 2026-06-21 07:54:13 on dist-test-slave-lbpt"
server_key: "31d624f9bf939eb2e02603c7642bde17"
server_key_iv: "c78b2cf35b548bbb84dc6e80589b9d27"
server_key_version: "kuduclusterkey@0"
I20260621 07:54:14.480079 25129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:14.525295 25129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:14.529083 25129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:14.529340 25129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:14.530081 25129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:14.531203 25129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:54:14.531312 25129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:14.531440 25129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:54:14.531517 25129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:14.542658 25129 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:38717
I20260621 07:54:14.543031 25288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:38717 every 8 connection(s)
I20260621 07:54:14.544122 25129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:54:14.545833 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 25129
I20260621 07:54:14.546342 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.20:58065/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:44905
--local_ip_for_outbound_sockets=127.12.12.2
--tserver_master_addrs=127.12.12.62:36199
--webserver_port=46581
--webserver_interface=127.12.12.2
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.12.20:58065/kms with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:54:14 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028454, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:14.572592 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:14.546803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:36199 (local address 127.12.12.1:53005)
0621 07:54:14.547562 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:14.547627 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:14.551248 (+ 3621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:14.553492 (+ 2244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:14.553503 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:14.553896 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:14.554672 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:14.554697 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:14.556877 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:14.556883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:14.558579 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:14.558595 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:14.560166 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:14.560170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:14.560364 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:14.562650 (+ 2286us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:14.562673 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:14.564505 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:14.568609 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:14.568616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:14.568636 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:14.569022 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:14.569480 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:14.569483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:14.569485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:14.569653 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:14.570522 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:14.570529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:14.570884 (+ 355us) client_negotiation.cc:769] Sending connection context
0621 07:54:14.571085 (+ 201us) client_negotiation.cc:240] Negotiation successful
0621 07:54:14.571470 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":140,"threads_started":1}
I20260621 07:54:14.575204 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:14.548407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:53005 (local address 127.12.12.62:36199)
0621 07:54:14.548869 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:14.548897 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:14.548958 (+ 61us) server_negotiation.cc:407] Connection header received
0621 07:54:14.551957 (+ 2999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:14.551977 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:14.552481 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:14.552878 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:14.554905 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:14.556682 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:14.558787 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:14.559734 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:14.559906 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:14.566034 (+ 6128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:14.566044 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:14.566072 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:14.566123 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:14.568404 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:14.569191 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:14.569200 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:14.569206 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:14.569369 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:14.569759 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:14.569762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:14.569764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:14.570286 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:14.570448 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:14.573811 (+ 3363us) server_negotiation.cc:299] Negotiation successful
0621 07:54:14.574128 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":135,"threads_started":1}
I20260621 07:54:14.578027 25289 heartbeater.cc:347] Connected to a master server at 127.12.12.62:36199
I20260621 07:54:14.578302 25289 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:14.578936 25289 heartbeater.cc:511] Master 127.12.12.62:36199 requested a full tablet report, sending...
W20260621 07:54:14.581455 25289 heartbeater.cc:644] Failed to heartbeat to 127.12.12.62:36199 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 07:54:14.581596 25289 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:14.581800 25289 heartbeater.cc:511] Master 127.12.12.62:36199 requested a full tablet report, sending...
I20260621 07:54:14.582947 25289 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:14.583159 25289 heartbeater.cc:511] Master 127.12.12.62:36199 requested a full tablet report, sending...
W20260621 07:54:14.583997 25289 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:54:14.681727 25292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:14.682088 25292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:14.682237 25292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:14.687091 25292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:14.687182 25292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:14.687273 25292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:54:14.693674 25292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.12.20:58065/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:44905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=46581
--tserver_master_addrs=127.12.12.62:36199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:14.695003 25292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:14.696116 25292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:14.708487 25298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:14.708787 25292 server_base.cc:1034] running on GCE node
W20260621 07:54:14.708478 25301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:14.710390 25299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:14.711148 25292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:14.712059 25292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:14.713315 25292 hybrid_clock.cc:648] HybridClock initialized: now 1782028454713275 us; error 59 us; skew 500 ppm
Jun 21 07:54:14 dist-test-slave-lbpt krb5kdc[24095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028454, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:14.720402 25292 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:54:14.721738 25292 webserver.cc:460] Webserver started at http://127.12.12.2:46581/ using document root <none> and password file <none>
I20260621 07:54:14.722574 25292 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:14.722721 25292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 07:54:14 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028454, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:54:14.744592 25292 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:54:14.750485 25309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:14.752388 25292 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260621 07:54:14.752573 25292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "ef0e9ecb18d44a7880988be012abe1fe"
format_stamp: "Formatted at 2026-06-21 07:54:13 on dist-test-slave-lbpt"
server_key: "4a63645bf99dc9875c7b3bfa3c607f5d"
server_key_iv: "c587c15d2f936f95dfdc316fd3286b84"
server_key_version: "kuduclusterkey@0"
I20260621 07:54:14.753194 25292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:14.775362 25292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:14.778913 25292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:14.779146 25292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:14.779814 25292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:14.780737 25292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:54:14.780807 25292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:14.780874 25292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:54:14.780927 25292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:14.792608 25292 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:44905
I20260621 07:54:14.792799 25423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:44905 every 8 connection(s)
I20260621 07:54:14.793779 25292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:54:14.797747 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 25292
I20260621 07:54:14.798273 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.20:58065/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:42347
--local_ip_for_outbound_sockets=127.12.12.3
--tserver_master_addrs=127.12.12.62:36199
--webserver_port=38925
--webserver_interface=127.12.12.3
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.12.20:58065/kms with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:54:14 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028454, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:14.810302 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:14.795861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:33601 (local address 127.12.12.62:36199)
0621 07:54:14.796017 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:14.796022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:14.797411 (+ 1389us) server_negotiation.cc:407] Connection header received
0621 07:54:14.798826 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:14.798833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:14.798905 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:14.799018 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:14.800788 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:14.801463 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:14.803077 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:14.803498 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:14.803603 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:14.806513 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:14.806517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:14.806520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:14.806556 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:14.808077 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:14.808632 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:14.808636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:14.808639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:14.808722 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:14.809078 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:14.809081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:14.809083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:14.809270 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:14.809373 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:14.809974 (+ 601us) server_negotiation.cc:299] Negotiation successful
0621 07:54:14.810126 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 07:54:14.811434 25426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:14.796425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:36199 (local address 127.12.12.2:33601)
0621 07:54:14.797042 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:14.797264 (+ 222us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:14.798522 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:14.799363 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:14.799373 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:14.799763 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:14.800566 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:14.800590 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:14.801607 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:14.801611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:14.802915 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:14.802928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:14.803829 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:14.803833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:14.804043 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:14.804763 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:14.804786 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:14.806316 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:14.808218 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:14.808225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:14.808242 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:14.808513 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:14.808821 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:14.808824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:14.808826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:14.808974 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:14.809381 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:14.809391 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:14.809727 (+ 336us) client_negotiation.cc:769] Sending connection context
0621 07:54:14.809920 (+ 193us) client_negotiation.cc:240] Negotiation successful
0621 07:54:14.810265 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":154,"threads_started":1}
I20260621 07:54:14.812852 25424 heartbeater.cc:347] Connected to a master server at 127.12.12.62:36199
I20260621 07:54:14.813158 25424 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:14.813671 25424 heartbeater.cc:511] Master 127.12.12.62:36199 requested a full tablet report, sending...
W20260621 07:54:14.815276 25424 heartbeater.cc:644] Failed to heartbeat to 127.12.12.62:36199 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 07:54:14.815415 25424 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:14.815603 25424 heartbeater.cc:511] Master 127.12.12.62:36199 requested a full tablet report, sending...
I20260621 07:54:14.816401 25424 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:14.816586 25424 heartbeater.cc:511] Master 127.12.12.62:36199 requested a full tablet report, sending...
W20260621 07:54:14.817389 25424 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:54:14.942194 25427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:14.942682 25427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:14.942808 25427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:14.949007 25427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:14.949170 25427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:14.949286 25427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:54:14.957327 25427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.12.20:58065/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:42347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=38925
--tserver_master_addrs=127.12.12.62:36199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:14.958837 25427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:14.960335 25427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:14.971418 25433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:14.971418 25432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:14.972393 25427 server_base.cc:1034] running on GCE node
W20260621 07:54:14.972517 25435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:14.972965 25427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:14.973697 25427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:14.974871 25427 hybrid_clock.cc:648] HybridClock initialized: now 1782028454974858 us; error 39 us; skew 500 ppm
Jun 21 07:54:14 dist-test-slave-lbpt krb5kdc[24095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028454, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:14.977962 25427 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:54:14.979095 25427 webserver.cc:460] Webserver started at http://127.12.12.3:38925/ using document root <none> and password file <none>
I20260621 07:54:14.980036 25427 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:14.980185 25427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 07:54:14 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028454, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:54:15.006538 25427 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 07:54:15.015395 25443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:15.017234 25427 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260621 07:54:15.017371 25427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "dbb2c727ab634a08942c32a89ff60e54"
format_stamp: "Formatted at 2026-06-21 07:54:13 on dist-test-slave-lbpt"
server_key: "3a8dfd96d2f5c1f4c1135024a7ddc736"
server_key_iv: "8144616b5a3dc538b1fbff079ea6e5c7"
server_key_version: "kuduclusterkey@0"
I20260621 07:54:15.018112 25427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:15.032384 25427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:15.038292 25427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:15.038620 25427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:15.039431 25427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:15.040566 25427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:54:15.040673 25427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:15.040778 25427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:54:15.040848 25427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:15.054220 25427 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:42347
I20260621 07:54:15.054276 25556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:42347 every 8 connection(s)
I20260621 07:54:15.056756 25427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:54:15.057166 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 25427
I20260621 07:54:15.068881 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:15.057962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34598 (local address 127.12.12.62:36199)
0621 07:54:15.058179 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:15.058187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:15.058522 (+ 335us) server_negotiation.cc:407] Connection header received
0621 07:54:15.058699 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:15.058703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:15.058761 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:15.058878 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:15.059923 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.060527 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.061967 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.062279 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.062445 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:15.064400 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:15.064404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:15.064407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:15.064446 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:15.066358 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:15.067097 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:15.067101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:15.067104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:15.067204 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:15.067612 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:15.067616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:15.067617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:15.067862 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:15.068058 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:15.068532 (+ 474us) server_negotiation.cc:299] Negotiation successful
0621 07:54:15.068680 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Jun 21 07:54:15 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028454, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:15.081115 25561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:15.062313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:36199 (local address 127.12.12.3:52407)
0621 07:54:15.062988 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:15.063063 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:15.064828 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:15.066500 (+ 1672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:15.066538 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:15.067109 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:15.068187 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:15.068236 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.069797 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.069801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:15.071639 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:15.071651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.072204 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.072210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:15.072420 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:15.073159 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:15.073181 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:15.074988 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:15.077210 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:15.077216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:15.077235 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:15.077623 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:15.077964 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:15.077968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:15.077970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:15.078175 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:15.078800 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:15.078807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:15.079225 (+ 418us) client_negotiation.cc:769] Sending connection context
0621 07:54:15.079480 (+ 255us) client_negotiation.cc:240] Negotiation successful
0621 07:54:15.079891 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":172,"threads_started":1}
I20260621 07:54:15.082779 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:15.061750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:52407 (local address 127.12.12.62:36199)
0621 07:54:15.062028 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:15.062038 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:15.063390 (+ 1352us) server_negotiation.cc:407] Connection header received
0621 07:54:15.065465 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:15.065469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:15.065524 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:15.065635 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:15.068447 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.069625 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.071760 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.072000 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.072136 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:15.075171 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:15.075175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:15.075177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:15.075218 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:15.077073 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:15.077736 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:15.077739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:15.077741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:15.077853 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:15.078311 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:15.078314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:15.078316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:15.078622 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:15.078736 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:15.081916 (+ 3180us) server_negotiation.cc:299] Negotiation successful
0621 07:54:15.082098 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":117,"threads_started":1}
I20260621 07:54:15.084441 25557 heartbeater.cc:347] Connected to a master server at 127.12.12.62:36199
I20260621 07:54:15.084723 25557 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:15.085448 25557 heartbeater.cc:511] Master 127.12.12.62:36199 requested a full tablet report, sending...
W20260621 07:54:15.087378 25557 heartbeater.cc:644] Failed to heartbeat to 127.12.12.62:36199 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 07:54:15.087529 25557 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:15.087715 25557 heartbeater.cc:511] Master 127.12.12.62:36199 requested a full tablet report, sending...
I20260621 07:54:15.089521 25091 ts_manager.cc:194] Registered new tserver with Master: dbb2c727ab634a08942c32a89ff60e54 (127.12.12.3:42347)
I20260621 07:54:15.091331 25091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:52407
Jun 21 07:54:15 dist-test-slave-lbpt krb5kdc[24095](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782028455, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:15.584230 25289 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:15.584558 25289 heartbeater.cc:511] Master 127.12.12.62:36199 requested a full tablet report, sending...
I20260621 07:54:15.585323 25091 ts_manager.cc:194] Registered new tserver with Master: 10e714f3b7174620bdfa444bd4f817bc (127.12.12.1:38717)
I20260621 07:54:15.586098 25091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:53005
Jun 21 07:54:15 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028455, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:54:15.817667 25424 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:15.817955 25424 heartbeater.cc:511] Master 127.12.12.62:36199 requested a full tablet report, sending...
I20260621 07:54:15.818706 25091 ts_manager.cc:194] Registered new tserver with Master: ef0e9ecb18d44a7880988be012abe1fe (127.12.12.2:44905)
I20260621 07:54:15.819476 25091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:33601
I20260621 07:54:15.828832 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:54:15.844280 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:15.831368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.12.12.62:36199)
0621 07:54:15.831961 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:15.831966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:15.832217 (+ 251us) server_negotiation.cc:407] Connection header received
0621 07:54:15.832430 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:15.832434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:15.832495 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:15.832564 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:15.833687 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.834505 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.835707 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.836061 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.836154 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:15.839632 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:15.839636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:15.839639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:15.839671 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:15.841735 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:15.842302 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:15.842306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:15.842308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:15.842433 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:15.843157 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:15.843160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:15.843162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:15.843427 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:15.843544 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:15.843939 (+ 395us) server_negotiation.cc:299] Negotiation successful
0621 07:54:15.844086 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":97,"threads_started":1}
I20260621 07:54:15.850327 25091 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:54:15.855212 25091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:54:15.882169 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:15.868855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:44905 (local address 127.0.0.1:39618)
0621 07:54:15.870682 (+ 1827us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:15.870735 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:15.870865 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:15.875434 (+ 4569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:15.875441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:15.875464 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:15.876480 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:15.876491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.877991 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.877994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:15.880212 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:15.880223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.881571 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.881575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:15.881721 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:15.881764 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 07:54:15.881876 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 07:54:15.882008 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1641,"thread_start_us":107,"threads_started":1}
I20260621 07:54:15.886842 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:15.868978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39618 (local address 127.12.12.2:44905)
0621 07:54:15.874968 (+ 5990us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:15.874974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:15.875006 (+ 32us) server_negotiation.cc:407] Connection header received
0621 07:54:15.875077 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:15.875081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:15.875273 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:15.875448 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:15.876639 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.877853 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.880349 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.881242 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.881350 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:15.881508 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:15.886502 (+ 4994us) server_negotiation.cc:299] Negotiation successful
0621 07:54:15.886649 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5864,"thread_start_us":76,"threads_started":1}
I20260621 07:54:15.890909 25587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:15.869160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:42347 (local address 127.0.0.1:50138)
0621 07:54:15.875722 (+ 6562us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:15.875739 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:15.875831 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:15.886492 (+ 10661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:15.886495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:15.886517 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:15.886602 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:15.886607 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.888032 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.888036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:15.889111 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:15.889123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.890472 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.890480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:15.890573 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:15.890599 (+ 26us) client_negotiation.cc:769] Sending connection context
0621 07:54:15.890675 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 07:54:15.890757 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6484,"thread_start_us":90,"threads_started":1}
I20260621 07:54:15.890312 25347 tablet_service.cc:1467] Processing CreateTablet for tablet 655a7727795445c7ac16c5f4b339f7d9 (DEFAULT_TABLE table=test-table [id=4f6390a4818847929ad9ddf5d6daaac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:54:15.891451 25588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:15.869417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.12.12.3:42347)
0621 07:54:15.874669 (+ 5252us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:15.874677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:15.875931 (+ 1254us) server_negotiation.cc:407] Connection header received
0621 07:54:15.876019 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:15.876024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:15.876248 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:15.876391 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:15.886784 (+ 10393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.887886 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.889284 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.890079 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.890200 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:15.890362 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:15.891191 (+ 829us) server_negotiation.cc:299] Negotiation successful
0621 07:54:15.891301 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5129,"thread_start_us":106,"threads_started":1}
I20260621 07:54:15.892786 25347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 655a7727795445c7ac16c5f4b339f7d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:15.894537 25491 tablet_service.cc:1467] Processing CreateTablet for tablet 655a7727795445c7ac16c5f4b339f7d9 (DEFAULT_TABLE table=test-table [id=4f6390a4818847929ad9ddf5d6daaac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:54:15.895686 25491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 655a7727795445c7ac16c5f4b339f7d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:15.904448 25590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:15.869918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60456 (local address 127.12.12.1:38717)
0621 07:54:15.870364 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:15.870372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:15.882663 (+ 12291us) server_negotiation.cc:407] Connection header received
0621 07:54:15.882900 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:15.882906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:15.883133 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:15.883256 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:15.883641 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.884870 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.902547 (+ 17677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.903496 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.903597 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:15.903776 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:15.904181 (+ 405us) server_negotiation.cc:299] Negotiation successful
0621 07:54:15.904291 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":100,"threads_started":1}
I20260621 07:54:15.904909 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:15.869616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:38717 (local address 127.0.0.1:60456)
0621 07:54:15.882571 (+ 12955us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:15.882585 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:15.882757 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:15.883439 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:15.883442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:15.883457 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:15.883530 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:15.883535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.885860 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.885863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:15.900548 (+ 14685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:15.900565 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:15.903902 (+ 3337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:15.903906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:15.904007 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:15.904031 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 07:54:15.904669 (+ 638us) client_negotiation.cc:240] Negotiation successful
0621 07:54:15.904782 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12897,"thread_start_us":113,"threads_started":1}
I20260621 07:54:15.908301 25223 tablet_service.cc:1467] Processing CreateTablet for tablet 655a7727795445c7ac16c5f4b339f7d9 (DEFAULT_TABLE table=test-table [id=4f6390a4818847929ad9ddf5d6daaac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:54:15.909505 25223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 655a7727795445c7ac16c5f4b339f7d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:15.909848 25591 tablet_bootstrap.cc:492] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54: Bootstrap starting.
I20260621 07:54:15.912776 25591 tablet_bootstrap.cc:654] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:15.913683 25591 log.cc:824] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:15.914523 25592 tablet_bootstrap.cc:492] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe: Bootstrap starting.
I20260621 07:54:15.916903 25592 tablet_bootstrap.cc:654] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:15.919030 25592 log.cc:824] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:15.920151 25594 tablet_bootstrap.cc:492] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc: Bootstrap starting.
I20260621 07:54:15.921317 25591 tablet_bootstrap.cc:492] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54: No bootstrap required, opened a new log
I20260621 07:54:15.921528 25591 ts_tablet_manager.cc:1397] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260621 07:54:15.922401 25594 tablet_bootstrap.cc:654] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:15.923317 25594 log.cc:824] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:15.925873 25594 tablet_bootstrap.cc:492] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc: No bootstrap required, opened a new log
I20260621 07:54:15.926081 25594 ts_tablet_manager.cc:1397] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:54:15.926177 25591 raft_consensus.cc:348] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e9ecb18d44a7880988be012abe1fe" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44905 } } peers { permanent_uuid: "10e714f3b7174620bdfa444bd4f817bc" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38717 } } peers { permanent_uuid: "dbb2c727ab634a08942c32a89ff60e54" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42347 } }
I20260621 07:54:15.926509 25591 raft_consensus.cc:374] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:15.926586 25591 raft_consensus.cc:729] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2c727ab634a08942c32a89ff60e54, State: Initialized, Role: FOLLOWER
I20260621 07:54:15.927109 25591 consensus_queue.cc:260] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e9ecb18d44a7880988be012abe1fe" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44905 } } peers { permanent_uuid: "10e714f3b7174620bdfa444bd4f817bc" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38717 } } peers { permanent_uuid: "dbb2c727ab634a08942c32a89ff60e54" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42347 } }
I20260621 07:54:15.928223 25591 ts_tablet_manager.cc:1428] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260621 07:54:15.928613 25557 heartbeater.cc:503] Master 127.12.12.62:36199 was elected leader, sending a full tablet report...
I20260621 07:54:15.930773 25592 tablet_bootstrap.cc:492] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe: No bootstrap required, opened a new log
I20260621 07:54:15.930996 25592 ts_tablet_manager.cc:1397] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20260621 07:54:15.934183 25594 raft_consensus.cc:348] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e9ecb18d44a7880988be012abe1fe" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44905 } } peers { permanent_uuid: "10e714f3b7174620bdfa444bd4f817bc" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38717 } } peers { permanent_uuid: "dbb2c727ab634a08942c32a89ff60e54" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42347 } }
I20260621 07:54:15.934518 25594 raft_consensus.cc:374] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:15.934583 25594 raft_consensus.cc:729] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e714f3b7174620bdfa444bd4f817bc, State: Initialized, Role: FOLLOWER
I20260621 07:54:15.935117 25594 consensus_queue.cc:260] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e9ecb18d44a7880988be012abe1fe" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44905 } } peers { permanent_uuid: "10e714f3b7174620bdfa444bd4f817bc" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38717 } } peers { permanent_uuid: "dbb2c727ab634a08942c32a89ff60e54" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42347 } }
I20260621 07:54:15.936039 25592 raft_consensus.cc:348] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e9ecb18d44a7880988be012abe1fe" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44905 } } peers { permanent_uuid: "10e714f3b7174620bdfa444bd4f817bc" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38717 } } peers { permanent_uuid: "dbb2c727ab634a08942c32a89ff60e54" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42347 } }
I20260621 07:54:15.936256 25594 ts_tablet_manager.cc:1428] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260621 07:54:15.936305 25592 raft_consensus.cc:374] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:15.936362 25592 raft_consensus.cc:729] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0e9ecb18d44a7880988be012abe1fe, State: Initialized, Role: FOLLOWER
I20260621 07:54:15.936618 25289 heartbeater.cc:503] Master 127.12.12.62:36199 was elected leader, sending a full tablet report...
I20260621 07:54:15.936828 25592 consensus_queue.cc:260] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e9ecb18d44a7880988be012abe1fe" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44905 } } peers { permanent_uuid: "10e714f3b7174620bdfa444bd4f817bc" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38717 } } peers { permanent_uuid: "dbb2c727ab634a08942c32a89ff60e54" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42347 } }
I20260621 07:54:15.937852 25592 ts_tablet_manager.cc:1428] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260621 07:54:15.938196 25424 heartbeater.cc:503] Master 127.12.12.62:36199 was elected leader, sending a full tablet report...
W20260621 07:54:16.045775 25425 tablet.cc:2369] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:54:16.048244 25290 tablet.cc:2369] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:54:16.059414 25558 tablet.cc:2369] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 07:54:16 dist-test-slave-lbpt krb5kdc[24095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028455, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for HTTP/127.12.12.20@KRBTEST.COM
I20260621 07:54:16.218847 25597 raft_consensus.cc:482] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:54:16.219131 25597 raft_consensus.cc:504] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e9ecb18d44a7880988be012abe1fe" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44905 } } peers { permanent_uuid: "10e714f3b7174620bdfa444bd4f817bc" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38717 } } peers { permanent_uuid: "dbb2c727ab634a08942c32a89ff60e54" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42347 } }
I20260621 07:54:16.220768 25597 leader_election.cc:288] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0e9ecb18d44a7880988be012abe1fe (127.12.12.2:44905), 10e714f3b7174620bdfa444bd4f817bc (127.12.12.1:38717)
I20260621 07:54:16.226166 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:16.221145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:56693 (local address 127.12.12.2:44905)
0621 07:54:16.221271 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:16.221276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:16.221942 (+ 666us) server_negotiation.cc:407] Connection header received
0621 07:54:16.222150 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:16.222153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:16.222204 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:16.222249 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:16.222672 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.223418 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.224962 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.225506 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.225594 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:16.225660 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:16.225956 (+ 296us) server_negotiation.cc:299] Negotiation successful
0621 07:54:16.226034 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 07:54:16.226831 25602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:16.221035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:44905 (local address 127.12.12.3:56693)
0621 07:54:16.221846 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:16.221862 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:16.222025 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:16.222380 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:16.222383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:16.222399 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:16.222549 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:16.222556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.223592 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.223595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:16.224820 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:16.224831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.225747 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.225750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:16.225831 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:16.225851 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 07:54:16.226603 (+ 752us) client_negotiation.cc:240] Negotiation successful
0621 07:54:16.226691 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":126,"threads_started":1}
I20260621 07:54:16.227679 25377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "655a7727795445c7ac16c5f4b339f7d9" candidate_uuid: "dbb2c727ab634a08942c32a89ff60e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0e9ecb18d44a7880988be012abe1fe" is_pre_election: true
I20260621 07:54:16.228039 25377 raft_consensus.cc:2454] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbb2c727ab634a08942c32a89ff60e54 in term 0.
I20260621 07:54:16.229398 25445 leader_election.cc:302] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dbb2c727ab634a08942c32a89ff60e54, ef0e9ecb18d44a7880988be012abe1fe; no voters:
I20260621 07:54:16.229754 25597 raft_consensus.cc:2781] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:54:16.229830 25597 raft_consensus.cc:482] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:54:16.229867 25597 raft_consensus.cc:3037] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:16.231416 25597 raft_consensus.cc:504] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e9ecb18d44a7880988be012abe1fe" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44905 } } peers { permanent_uuid: "10e714f3b7174620bdfa444bd4f817bc" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38717 } } peers { permanent_uuid: "dbb2c727ab634a08942c32a89ff60e54" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42347 } }
I20260621 07:54:16.231799 25597 leader_election.cc:288] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [CANDIDATE]: Term 1 election: Requested vote from peers ef0e9ecb18d44a7880988be012abe1fe (127.12.12.2:44905), 10e714f3b7174620bdfa444bd4f817bc (127.12.12.1:38717)
I20260621 07:54:16.232337 25377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "655a7727795445c7ac16c5f4b339f7d9" candidate_uuid: "dbb2c727ab634a08942c32a89ff60e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0e9ecb18d44a7880988be012abe1fe"
I20260621 07:54:16.232466 25377 raft_consensus.cc:3037] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:16.233729 25377 raft_consensus.cc:2454] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbb2c727ab634a08942c32a89ff60e54 in term 1.
I20260621 07:54:16.234122 25445 leader_election.cc:302] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dbb2c727ab634a08942c32a89ff60e54, ef0e9ecb18d44a7880988be012abe1fe; no voters:
I20260621 07:54:16.234334 25597 raft_consensus.cc:2781] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:54:16.234586 25597 raft_consensus.cc:686] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [term 1 LEADER]: Becoming Leader. State: Replica: dbb2c727ab634a08942c32a89ff60e54, State: Running, Role: LEADER
I20260621 07:54:16.234966 25597 consensus_queue.cc:237] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e9ecb18d44a7880988be012abe1fe" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44905 } } peers { permanent_uuid: "10e714f3b7174620bdfa444bd4f817bc" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38717 } } peers { permanent_uuid: "dbb2c727ab634a08942c32a89ff60e54" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42347 } }
I20260621 07:54:16.238852 25598 raft_consensus.cc:482] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:54:16.239054 25598 raft_consensus.cc:504] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e9ecb18d44a7880988be012abe1fe" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44905 } } peers { permanent_uuid: "10e714f3b7174620bdfa444bd4f817bc" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38717 } } peers { permanent_uuid: "dbb2c727ab634a08942c32a89ff60e54" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42347 } }
I20260621 07:54:16.238747 25090 catalog_manager.cc:5495] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to dbb2c727ab634a08942c32a89ff60e54 (127.12.12.3). New cstate: current_term: 1 leader_uuid: "dbb2c727ab634a08942c32a89ff60e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e9ecb18d44a7880988be012abe1fe" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10e714f3b7174620bdfa444bd4f817bc" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbb2c727ab634a08942c32a89ff60e54" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 42347 } health_report { overall_health: HEALTHY } } }
I20260621 07:54:16.240289 25598 leader_election.cc:288] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0e9ecb18d44a7880988be012abe1fe (127.12.12.2:44905), dbb2c727ab634a08942c32a89ff60e54 (127.12.12.3:42347)
I20260621 07:54:16.243151 25590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:16.221555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:51539 (local address 127.12.12.1:38717)
0621 07:54:16.221666 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:16.221670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:16.227500 (+ 5830us) server_negotiation.cc:407] Connection header received
0621 07:54:16.227693 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:16.227696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:16.227748 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:16.227791 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:16.228276 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.229014 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.239309 (+ 10295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.239884 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.239975 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:16.240043 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:16.242940 (+ 2897us) server_negotiation.cc:299] Negotiation successful
0621 07:54:16.243035 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 07:54:16.243721 25602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:16.221436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:38717 (local address 127.12.12.3:51539)
0621 07:54:16.227427 (+ 5991us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:16.227440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:16.227582 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:16.227991 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:16.227993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:16.228009 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:16.228078 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:16.228082 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.237990 (+ 9908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.237994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:16.239133 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:16.239147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.242691 (+ 3544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.242694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:16.242796 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:16.242822 (+ 26us) client_negotiation.cc:769] Sending connection context
0621 07:54:16.243487 (+ 665us) client_negotiation.cc:240] Negotiation successful
0621 07:54:16.243592 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5934,"thread_start_us":97,"threads_started":1}
I20260621 07:54:16.245258 25240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "655a7727795445c7ac16c5f4b339f7d9" candidate_uuid: "dbb2c727ab634a08942c32a89ff60e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e714f3b7174620bdfa444bd4f817bc" is_pre_election: true
I20260621 07:54:16.245497 25240 raft_consensus.cc:2454] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbb2c727ab634a08942c32a89ff60e54 in term 0.
I20260621 07:54:16.246024 25241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "655a7727795445c7ac16c5f4b339f7d9" candidate_uuid: "dbb2c727ab634a08942c32a89ff60e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e714f3b7174620bdfa444bd4f817bc"
I20260621 07:54:16.246143 25241 raft_consensus.cc:3037] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:16.247516 25241 raft_consensus.cc:2454] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbb2c727ab634a08942c32a89ff60e54 in term 1.
I20260621 07:54:16.255748 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:16.241505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:41797 (local address 127.12.12.2:44905)
0621 07:54:16.241765 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:16.241770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:16.251291 (+ 9521us) server_negotiation.cc:407] Connection header received
0621 07:54:16.251495 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:16.251498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:16.251551 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:16.251603 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:16.251989 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.252822 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.254452 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.255060 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.255133 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:16.255202 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:16.255526 (+ 324us) server_negotiation.cc:299] Negotiation successful
0621 07:54:16.255612 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260621 07:54:16.256229 25607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:16.240682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:44905 (local address 127.12.12.1:41797)
0621 07:54:16.251186 (+ 10504us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:16.251203 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:16.251377 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:16.251747 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:16.251750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:16.251767 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:16.251877 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:16.251882 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.252963 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.252967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:16.254284 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:16.254295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.255294 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.255298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:16.255400 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:16.255422 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:54:16.255982 (+ 560us) client_negotiation.cc:240] Negotiation successful
0621 07:54:16.256090 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10416,"thread_start_us":112,"threads_started":1}
I20260621 07:54:16.256879 25377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "655a7727795445c7ac16c5f4b339f7d9" candidate_uuid: "10e714f3b7174620bdfa444bd4f817bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0e9ecb18d44a7880988be012abe1fe" is_pre_election: true
I20260621 07:54:16.257143 25377 raft_consensus.cc:2379] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10e714f3b7174620bdfa444bd4f817bc in current term 1: Already voted for candidate dbb2c727ab634a08942c32a89ff60e54 in this term.
I20260621 07:54:16.260612 25588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:16.241631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:42285 (local address 127.12.12.3:42347)
0621 07:54:16.241809 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:16.241814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:16.256661 (+ 14847us) server_negotiation.cc:407] Connection header received
0621 07:54:16.256849 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:16.256852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:16.256911 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:16.256960 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:16.257346 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.258049 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.259401 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.259992 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.260074 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:16.260132 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:16.260423 (+ 291us) server_negotiation.cc:299] Negotiation successful
0621 07:54:16.260497 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260621 07:54:16.260999 25607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:16.241081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:42347 (local address 127.12.12.1:42285)
0621 07:54:16.256588 (+ 15507us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:16.256603 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:16.256732 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:16.257126 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:16.257128 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:16.257145 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:16.257225 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:16.257230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.258180 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.258182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:16.259270 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:16.259280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.260215 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.260219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:16.260303 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:16.260324 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:54:16.260802 (+ 478us) client_negotiation.cc:240] Negotiation successful
0621 07:54:16.260878 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15428,"thread_start_us":91,"threads_started":1}
I20260621 07:54:16.261684 25495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "655a7727795445c7ac16c5f4b339f7d9" candidate_uuid: "10e714f3b7174620bdfa444bd4f817bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2c727ab634a08942c32a89ff60e54" is_pre_election: true
I20260621 07:54:16.262497 25176 leader_election.cc:302] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10e714f3b7174620bdfa444bd4f817bc; no voters: dbb2c727ab634a08942c32a89ff60e54, ef0e9ecb18d44a7880988be012abe1fe
I20260621 07:54:16.263005 25598 raft_consensus.cc:2726] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260621 07:54:16.339867 25588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:16.334682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.12.12.3:42347)
0621 07:54:16.334813 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:16.334818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:16.334832 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:54:16.334923 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:16.334927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:16.334995 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:16.335103 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:54:16.335521 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.336332 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.337641 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:16.337987 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:16.338086 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:16.338537 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:54:16.339105 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:54:16.339232 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:16.339649 (+ 417us) server_negotiation.cc:299] Negotiation successful
0621 07:54:16.339714 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:54:16.348009 25241 raft_consensus.cc:1261] T 655a7727795445c7ac16c5f4b339f7d9 P 10e714f3b7174620bdfa444bd4f817bc [term 1 FOLLOWER]: Refusing update from remote peer dbb2c727ab634a08942c32a89ff60e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:54:16.348923 25377 raft_consensus.cc:1261] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe [term 1 FOLLOWER]: Refusing update from remote peer dbb2c727ab634a08942c32a89ff60e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:54:16.349858 25604 consensus_queue.cc:1035] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0e9ecb18d44a7880988be012abe1fe" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:54:16.351107 25604 consensus_queue.cc:1035] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10e714f3b7174620bdfa444bd4f817bc" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 38717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:54:16.375295 25609 mvcc.cc:205] Tried to move back new op lower bound from 7299188557191540736 to 7299188556742017024. Current Snapshot: MvccSnapshot[applied={T|T < 7299188557191540736}]
I20260621 07:54:16.380829 25612 mvcc.cc:205] Tried to move back new op lower bound from 7299188557191540736 to 7299188556742017024. Current Snapshot: MvccSnapshot[applied={T|T < 7299188557191540736}]
I20260621 07:54:16.385020 25610 mvcc.cc:205] Tried to move back new op lower bound from 7299188557191540736 to 7299188556742017024. Current Snapshot: MvccSnapshot[applied={T|T < 7299188557191540736}]
I20260621 07:54:16.395247 25090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:54:16.395454 25090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:54:16.398380 25090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e10998b06b445138307dd67d8a406b8: Sending DeleteTablet for 3 replicas of tablet 655a7727795445c7ac16c5f4b339f7d9
I20260621 07:54:16.400113 25347 tablet_service.cc:1514] Processing DeleteTablet for tablet 655a7727795445c7ac16c5f4b339f7d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:54:16 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:39618
I20260621 07:54:16.400696 25223 tablet_service.cc:1514] Processing DeleteTablet for tablet 655a7727795445c7ac16c5f4b339f7d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:54:16 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:60456
I20260621 07:54:16.401167 25491 tablet_service.cc:1514] Processing DeleteTablet for tablet 655a7727795445c7ac16c5f4b339f7d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:54:16 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:50138
I20260621 07:54:16.402606 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 25129
I20260621 07:54:16.412467 25622 tablet_replica.cc:330] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54: stopping tablet replica
I20260621 07:54:16.412774 25620 tablet_replica.cc:330] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe: stopping tablet replica
I20260621 07:54:16.413247 25620 raft_consensus.cc:2229] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:54:16.413628 25622 raft_consensus.cc:2229] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:54:16.414156 25622 raft_consensus.cc:2258] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:54:16.414604 25620 raft_consensus.cc:2258] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:54:16.416927 25620 ts_tablet_manager.cc:1905] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:54:16.417972 25622 ts_tablet_manager.cc:1905] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:54:16.421046 25622 ts_tablet_manager.cc:1918] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:54:16.424515 25620 ts_tablet_manager.cc:1918] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:54:16.424741 25620 log.cc:1196] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/655a7727795445c7ac16c5f4b339f7d9
I20260621 07:54:16.425215 25620 ts_tablet_manager.cc:1939] T 655a7727795445c7ac16c5f4b339f7d9 P ef0e9ecb18d44a7880988be012abe1fe: Deleting consensus metadata
I20260621 07:54:16.427034 25622 log.cc:1196] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/655a7727795445c7ac16c5f4b339f7d9
W20260621 07:54:16.427304 25076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.1:38717 (error 108)
I20260621 07:54:16.427402 25622 ts_tablet_manager.cc:1939] T 655a7727795445c7ac16c5f4b339f7d9 P dbb2c727ab634a08942c32a89ff60e54: Deleting consensus metadata
I20260621 07:54:16.428292 25076 catalog_manager.cc:4841] TS ef0e9ecb18d44a7880988be012abe1fe (127.12.12.2:44905): tablet 655a7727795445c7ac16c5f4b339f7d9 (table test-table [id=4f6390a4818847929ad9ddf5d6daaac0]) successfully deleted
I20260621 07:54:16.428637 25075 catalog_manager.cc:4841] TS dbb2c727ab634a08942c32a89ff60e54 (127.12.12.3:42347): tablet 655a7727795445c7ac16c5f4b339f7d9 (table test-table [id=4f6390a4818847929ad9ddf5d6daaac0]) successfully deleted
I20260621 07:54:16.428675 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 25292
I20260621 07:54:16.434942 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 25427
I20260621 07:54:16.435457 25627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:16.430893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:38717 (local address 127.0.0.1:60468)
0621 07:54:16.435022 (+ 4129us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:16.435113 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:38717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4033,"thread_start_us":55,"threads_started":1}
W20260621 07:54:16.435768 25076 catalog_manager.cc:4584] TS 10e714f3b7174620bdfa444bd4f817bc (127.12.12.1:38717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 655a7727795445c7ac16c5f4b339f7d9: Network error: Client connection negotiation failed: client connection to 127.12.12.1:38717: connect: Connection refused (error 111)
I20260621 07:54:16.440781 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 25058
I20260621 07:54:16.446520 12336 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 07:54:16.791832 12336 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 07:54:16.791951 12336 mini_ranger.cc:66] Stopping Ranger...
I20260621 07:54:17.137419 12336 mini_ranger.cc:68] Stopped Ranger
2026-06-21 07:54:17.137 UTC [24125] LOG: received smart shutdown request
2026-06-21 07:54:17.141 UTC [24125] LOG: background worker "logical replication launcher" (PID 24134) exited with exit code 1
2026-06-21 07:54:17.141 UTC [24129] LOG: shutting down
2026-06-21 07:54:17.289 UTC [24125] LOG: database system is shut down
2026-06-21T07:54:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49388 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:54:17 dist-test-slave-lbpt krb5kdc[25638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:54:17 dist-test-slave-lbpt krb5kdc[25638](info): set up 2 sockets
Jun 21 07:54:17 dist-test-slave-lbpt krb5kdc[25638](info): commencing operation
krb5kdc: starting...
W20260621 07:54:18.450625 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:54:18 dist-test-slave-lbpt krb5kdc[25638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:54:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:54:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:54:18.573132 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:34939
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:33119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:34939,127.12.12.61:40213,127.12.12.60:43037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:54:18.686231 25651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:18.686534 25651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:18.686620 25651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:18.690663 25651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:54:18.690721 25651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:18.690740 25651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:18.690790 25651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:54:18.690825 25651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:54:18.696079 25651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:33119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:34939,127.12.12.61:40213,127.12.12.60:43037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:34939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:18.697220 25651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:18.698334 25651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:18.707387 25657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:18.707382 25656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:18.707377 25659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:18.707674 25651 server_base.cc:1034] running on GCE node
I20260621 07:54:18.708357 25651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:18.709560 25651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:18.710757 25651 hybrid_clock.cc:648] HybridClock initialized: now 1782028458710719 us; error 48 us; skew 500 ppm
Jun 21 07:54:18 dist-test-slave-lbpt krb5kdc[25638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028458, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:18.713760 25651 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:54:18.714807 25651 webserver.cc:460] Webserver started at http://127.12.12.62:37887/ using document root <none> and password file <none>
I20260621 07:54:18.715387 25651 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:18.715479 25651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:18.715739 25651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:54:18.717882 25651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "54ecb14d607a49a59c9c9754d8a08187"
format_stamp: "Formatted at 2026-06-21 07:54:18 on dist-test-slave-lbpt"
server_key: "4049896a7894bfc87c04ff22cdcf217d"
server_key_iv: "8060a34dc4244bea36ee94b5ab0bc178"
server_key_version: "encryptionkey@0"
I20260621 07:54:18.718535 25651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "54ecb14d607a49a59c9c9754d8a08187"
format_stamp: "Formatted at 2026-06-21 07:54:18 on dist-test-slave-lbpt"
server_key: "4049896a7894bfc87c04ff22cdcf217d"
server_key_iv: "8060a34dc4244bea36ee94b5ab0bc178"
server_key_version: "encryptionkey@0"
I20260621 07:54:18.722342 25651 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:54:18.724951 25666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:18.726300 25651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:54:18.726486 25651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "54ecb14d607a49a59c9c9754d8a08187"
format_stamp: "Formatted at 2026-06-21 07:54:18 on dist-test-slave-lbpt"
server_key: "4049896a7894bfc87c04ff22cdcf217d"
server_key_iv: "8060a34dc4244bea36ee94b5ab0bc178"
server_key_version: "encryptionkey@0"
I20260621 07:54:18.726621 25651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:18.747138 25651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:18.750844 25651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:18.751086 25651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:18.758754 25651 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:34939
I20260621 07:54:18.758814 25718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:34939 every 8 connection(s)
I20260621 07:54:18.759829 25651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:54:18.760095 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 25651
I20260621 07:54:18.760241 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:54:18.760596 12336 external_mini_cluster.cc:1428] Setting key 6a63a34052be95e2562ed508e7e50b57
I20260621 07:54:18.763303 25719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
WARNING: no policy specified for kudu/127.12.12.61@KRBTEST.COM; defaulting to no policy
I20260621 07:54:18.769043 25719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.61@KRBTEST.COM" created.
Jun 21 07:54:18 dist-test-slave-lbpt krb5kdc[25638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028458, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:18.787906 25722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:18.771920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59230 (local address 127.12.12.62:34939)
0621 07:54:18.772393 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:18.772412 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:18.772864 (+ 452us) server_negotiation.cc:407] Connection header received
0621 07:54:18.774078 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:18.774088 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:18.774335 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:18.774655 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:18.775974 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:18.777016 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:18.778970 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:18.779403 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:18.779538 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:18.782550 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:18.782556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:18.782568 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:18.782591 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:18.784419 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:18.784945 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:18.784953 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:18.784961 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:18.785064 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:18.785424 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:18.785428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:18.785429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:18.785880 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:18.786038 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:18.786457 (+ 419us) server_negotiation.cc:299] Negotiation successful
0621 07:54:18.786772 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":329,"spinlock_wait_cycles":22400,"thread_start_us":163,"threads_started":1}
I20260621 07:54:18.787915 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:18.771920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:34939 (local address 127.0.0.1:59230)
0621 07:54:18.772666 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:18.772724 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:18.773700 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:18.774714 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:18.774720 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:18.775326 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:18.775767 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:18.775792 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:18.777176 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:18.777180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:18.778792 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:18.778810 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:18.779604 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:18.779609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:18.779757 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:18.780449 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:18.780473 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:18.782323 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:18.784539 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:18.784549 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:18.784561 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:18.784831 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:18.785174 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:18.785178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:18.785180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:18.785297 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:18.786030 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:18.786037 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:18.786181 (+ 144us) client_negotiation.cc:769] Sending connection context
0621 07:54:18.786358 (+ 177us) client_negotiation.cc:240] Negotiation successful
0621 07:54:18.786823 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":100,"thread_start_us":199,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.61.keytab.
Entry for principal kudu/127.12.12.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.61.keytab.
I20260621 07:54:18.789851 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.61:40213
--webserver_interface=127.12.12.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.61.keytab
--principal=kudu/127.12.12.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:33119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:34939,127.12.12.61:40213,127.12.12.60:43037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:54:18.790786 25719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:54:18.791709 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:18.791349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40213 (local address 127.0.0.1:54892)
0621 07:54:18.791522 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:18.791617 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260621 07:54:18.791946 25667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40213: connect: Connection refused (error 111)
I20260621 07:54:18.792958 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:18.792648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40213 (local address 127.0.0.1:54906)
0621 07:54:18.792784 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:18.792887 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260621 07:54:18.793164 25719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.61:40213: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40213: connect: Connection refused (error 111)
I20260621 07:54:18.842518 25719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } attempt: 1
I20260621 07:54:18.843530 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:18.843164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40213 (local address 127.0.0.1:54918)
0621 07:54:18.843378 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:18.843445 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260621 07:54:18.844393 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:18.844140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40213 (local address 127.0.0.1:54932)
0621 07:54:18.844267 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:18.844328 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260621 07:54:18.844542 25719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.61:40213: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40213: connect: Connection refused (error 111)
W20260621 07:54:18.907938 25725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:18.908286 25725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:18.908380 25725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:18.912611 25725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:54:18.912704 25725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:18.912739 25725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:18.912775 25725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:54:18.912815 25725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:54:18.912722 25719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } attempt: 2
I20260621 07:54:18.913626 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:18.913323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40213 (local address 127.0.0.1:54946)
0621 07:54:18.913477 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:18.913549 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260621 07:54:18.914333 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:18.914123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40213 (local address 127.0.0.1:54952)
0621 07:54:18.914230 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:18.914284 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 07:54:18.914471 25719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.61:40213: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40213: connect: Connection refused (error 111)
I20260621 07:54:18.918658 25725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:33119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:34939,127.12.12.61:40213,127.12.12.60:43037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.61.keytab
--principal=kudu/127.12.12.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.61:40213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.12.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:18.919798 25725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:18.920857 25725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:18.929802 25732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:18.929863 25731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:18.930001 25725 server_base.cc:1034] running on GCE node
W20260621 07:54:18.929802 25734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:18.930848 25725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:18.932027 25725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:18.933247 25725 hybrid_clock.cc:648] HybridClock initialized: now 1782028458933216 us; error 48 us; skew 500 ppm
Jun 21 07:54:18 dist-test-slave-lbpt krb5kdc[25638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028458, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:18.936239 25725 init.cc:377] Logged in from keytab as kudu/127.12.12.61@KRBTEST.COM (short username kudu)
I20260621 07:54:18.937214 25725 webserver.cc:460] Webserver started at http://127.12.12.61:32849/ using document root <none> and password file <none>
I20260621 07:54:18.937808 25725 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:18.937902 25725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:18.938144 25725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:54:18.940327 25725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/data/instance:
uuid: "41835fabb4cb42f0a5e0b93f130e89ba"
format_stamp: "Formatted at 2026-06-21 07:54:18 on dist-test-slave-lbpt"
server_key: "0207fb8dd9860b0c7124d9bede77f202"
server_key_iv: "189210e19ba508c83eeaf7ecdd094b78"
server_key_version: "encryptionkey@0"
I20260621 07:54:18.940925 25725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/wal/instance:
uuid: "41835fabb4cb42f0a5e0b93f130e89ba"
format_stamp: "Formatted at 2026-06-21 07:54:18 on dist-test-slave-lbpt"
server_key: "0207fb8dd9860b0c7124d9bede77f202"
server_key_iv: "189210e19ba508c83eeaf7ecdd094b78"
server_key_version: "encryptionkey@0"
I20260621 07:54:18.944818 25725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260621 07:54:18.947417 25741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:18.948800 25725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 07:54:18.948975 25725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/wal
uuid: "41835fabb4cb42f0a5e0b93f130e89ba"
format_stamp: "Formatted at 2026-06-21 07:54:18 on dist-test-slave-lbpt"
server_key: "0207fb8dd9860b0c7124d9bede77f202"
server_key_iv: "189210e19ba508c83eeaf7ecdd094b78"
server_key_version: "encryptionkey@0"
I20260621 07:54:18.949100 25725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:18.965551 25725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:18.969049 25725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:18.969275 25725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:18.977102 25725 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.61:40213
I20260621 07:54:18.977133 25793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.61:40213 every 8 connection(s)
I20260621 07:54:18.978107 25725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/data/info.pb
I20260621 07:54:18.980929 25794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:18.984648 25794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:54:18.987744 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 25725
I20260621 07:54:18.987886 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-1/wal/instance
I20260621 07:54:18.988165 12336 external_mini_cluster.cc:1428] Setting key 282dd1a7f3ac21265b0ef394f45dd828
Jun 21 07:54:18 dist-test-slave-lbpt krb5kdc[25638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028458, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.61@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.60@KRBTEST.COM" created.
I20260621 07:54:19.001017 25722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:18.986530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.12.12.62:34939)
0621 07:54:18.986699 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:18.986703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:18.987685 (+ 982us) server_negotiation.cc:407] Connection header received
0621 07:54:18.988774 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:18.988778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:18.988840 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:18.988948 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:18.990326 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:18.990947 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:18.992827 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:18.993145 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:18.993243 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:18.996488 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:18.996492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:18.996494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:18.996524 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:18.998097 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:18.998770 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:18.998774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:18.998775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:18.998886 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:18.999395 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:18.999398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:18.999399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:18.999727 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:18.999819 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:19.000585 (+ 766us) server_negotiation.cc:299] Negotiation successful
0621 07:54:19.000757 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:54:19.001701 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:18.986950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:34939 (local address 127.0.0.1:59238)
0621 07:54:18.987450 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:18.987513 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:18.988559 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:18.989128 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:18.989136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:18.989479 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:18.990058 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:18.990071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:18.991104 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:18.991111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:18.992656 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:18.992673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:18.993334 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:18.993341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:18.993588 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:18.994302 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:18.994332 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:18.996278 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:18.998252 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:18.998258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:18.998273 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:18.998660 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:18.999046 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:18.999052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:18.999058 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:18.999260 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:18.999867 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:18.999879 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:19.000251 (+ 372us) client_negotiation.cc:769] Sending connection context
0621 07:54:19.000437 (+ 186us) client_negotiation.cc:240] Negotiation successful
0621 07:54:19.000742 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":155,"threads_started":1}
I20260621 07:54:19.003563 25794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:54:19.005640 25719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } attempt: 3
Jun 21 07:54:19 dist-test-slave-lbpt krb5kdc[25638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028458, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.61@KRBTEST.COM for kudu/127.12.12.61@KRBTEST.COM
Jun 21 07:54:19 dist-test-slave-lbpt krb5kdc[25638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028458, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for kudu/127.12.12.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.60.keytab.
Entry for principal kudu/127.12.12.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.60.keytab.
I20260621 07:54:19.017760 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.006077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40213 (local address 127.0.0.1:54970)
0621 07:54:19.006260 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.006279 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:19.006358 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:19.006836 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:19.006839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:19.007021 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:19.007140 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:19.007147 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.008339 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.008344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:19.009595 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:19.009607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.010182 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.010186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:19.010273 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.010791 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:19.010804 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:19.012971 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:19.015374 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:19.015379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:19.015382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:19.015672 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:19.016085 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:19.016088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:19.016089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:19.016231 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:19.016966 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:19.016972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:19.017135 (+ 163us) client_negotiation.cc:769] Sending connection context
0621 07:54:19.017262 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 07:54:19.017477 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260621 07:54:19.017748 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.004197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40213 (local address 127.0.0.1:54962)
0621 07:54:19.004330 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.004346 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:19.004429 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:19.004986 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:19.004989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:19.005176 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:19.005266 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:19.005272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.006751 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.006757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:19.007507 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:19.007516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.008090 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.008093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:19.008178 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.008799 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:19.008818 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:19.012457 (+ 3639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:19.015554 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:19.015557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:19.015558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:19.015871 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:19.016171 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:19.016175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:19.016176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:19.016264 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:19.017176 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:19.017180 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:19.017283 (+ 103us) client_negotiation.cc:769] Sending connection context
0621 07:54:19.017388 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 07:54:19.017564 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":44}
I20260621 07:54:19.017968 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.006139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.12.12.61:40213)
0621 07:54:19.006538 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:19.006550 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:19.006569 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:54:19.006610 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:19.006616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:19.006668 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:19.006816 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:19.007276 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.008241 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.009719 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.009971 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.010135 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.013179 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:19.013183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:19.013185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:19.013207 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:19.015234 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.015846 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.015851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.015852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.015952 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.016475 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.016478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.016480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.016802 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:19.016914 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:19.017420 (+ 506us) server_negotiation.cc:299] Negotiation successful
0621 07:54:19.017640 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":315,"thread_start_us":148,"threads_started":1}
I20260621 07:54:19.018725 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.60:43037
--webserver_interface=127.12.12.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.60.keytab
--principal=kudu/127.12.12.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:33119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:34939,127.12.12.61:40213,127.12.12.60:43037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:54:19.018767 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.004306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.12.12.61:40213)
0621 07:54:19.004642 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:19.004653 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:19.004671 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:54:19.004707 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:19.004711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:19.004867 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:19.005014 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:19.005376 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.006643 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.007615 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.007977 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.008096 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.012668 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:19.012691 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:19.012709 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:19.012753 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:19.015424 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.015972 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.015978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.015982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.016064 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.016942 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.016944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.016945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.017077 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:19.017149 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:19.017450 (+ 301us) server_negotiation.cc:299] Negotiation successful
0621 07:54:19.018615 (+ 1165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":973,"server-negotiator.queue_time_us":251,"thread_start_us":134,"threads_started":1}
I20260621 07:54:19.020234 25719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:54:19.020489 25794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:54:19.021059 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.020790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:43037 (local address 127.0.0.1:47362)
0621 07:54:19.020930 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.020987 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260621 07:54:19.021186 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.020899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:43037 (local address 127.0.0.1:47372)
0621 07:54:19.021050 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.021127 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260621 07:54:19.021517 25744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
I20260621 07:54:19.021900 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.021565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:43037 (local address 127.0.0.1:47382)
0621 07:54:19.021758 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.021816 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260621 07:54:19.022130 25719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:43037: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
I20260621 07:54:19.022365 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.022136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:43037 (local address 127.0.0.1:47392)
0621 07:54:19.022268 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.022313 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260621 07:54:19.022547 25794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:43037: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
I20260621 07:54:19.053315 25719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } attempt: 1
I20260621 07:54:19.054369 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.054013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:43037 (local address 127.0.0.1:47404)
0621 07:54:19.054183 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.054265 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260621 07:54:19.055158 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.054925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:43037 (local address 127.0.0.1:47408)
0621 07:54:19.055042 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.055094 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 07:54:19.055354 25719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:43037: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
I20260621 07:54:19.071810 25794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } attempt: 1
I20260621 07:54:19.072758 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.072442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:43037 (local address 127.0.0.1:47412)
0621 07:54:19.072613 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.072678 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260621 07:54:19.073485 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.073275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:43037 (local address 127.0.0.1:47424)
0621 07:54:19.073383 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.073436 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 07:54:19.073936 25794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:43037: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
I20260621 07:54:19.130620 25719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } attempt: 2
I20260621 07:54:19.131594 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.131194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:43037 (local address 127.0.0.1:47430)
0621 07:54:19.131417 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.131506 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260621 07:54:19.132407 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.132207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:43037 (local address 127.0.0.1:47436)
0621 07:54:19.132308 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.132355 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260621 07:54:19.132638 25719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:43037: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
W20260621 07:54:19.135708 25801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:19.135963 25801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:19.136005 25801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:19.140312 25801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:54:19.140380 25801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:19.140400 25801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:19.140419 25801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:54:19.140434 25801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:54:19.142122 25794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } attempt: 2
I20260621 07:54:19.142990 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.142696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:43037 (local address 127.0.0.1:47450)
0621 07:54:19.142843 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.142915 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260621 07:54:19.143750 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.143531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:43037 (local address 127.0.0.1:47460)
0621 07:54:19.143646 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.143699 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260621 07:54:19.143920 25794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:43037: Network error: Client connection negotiation failed: client connection to 127.12.12.60:43037: connect: Connection refused (error 111)
I20260621 07:54:19.146378 25801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:33119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:34939,127.12.12.61:40213,127.12.12.60:43037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.60.keytab
--principal=kudu/127.12.12.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.60:43037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.12.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:19.147517 25801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:19.148501 25801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:19.158069 25808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:19.158077 25810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:19.158061 25807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:19.158128 25801 server_base.cc:1034] running on GCE node
I20260621 07:54:19.159026 25801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:19.160199 25801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:19.161374 25801 hybrid_clock.cc:648] HybridClock initialized: now 1782028459161357 us; error 32 us; skew 500 ppm
Jun 21 07:54:19 dist-test-slave-lbpt krb5kdc[25638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028459, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:19.164291 25801 init.cc:377] Logged in from keytab as kudu/127.12.12.60@KRBTEST.COM (short username kudu)
I20260621 07:54:19.165158 25801 webserver.cc:460] Webserver started at http://127.12.12.60:45229/ using document root <none> and password file <none>
I20260621 07:54:19.165743 25801 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:19.165805 25801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:19.165977 25801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:54:19.168052 25801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/data/instance:
uuid: "fe047c319ae343a19d117e47a25b2b8c"
format_stamp: "Formatted at 2026-06-21 07:54:19 on dist-test-slave-lbpt"
server_key: "bea8d08cfcda134a7c93d21806cc0d73"
server_key_iv: "b421dd1b27f9edaa3e5174752f535859"
server_key_version: "encryptionkey@0"
I20260621 07:54:19.168548 25801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/wal/instance:
uuid: "fe047c319ae343a19d117e47a25b2b8c"
format_stamp: "Formatted at 2026-06-21 07:54:19 on dist-test-slave-lbpt"
server_key: "bea8d08cfcda134a7c93d21806cc0d73"
server_key_iv: "b421dd1b27f9edaa3e5174752f535859"
server_key_version: "encryptionkey@0"
I20260621 07:54:19.172421 25801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260621 07:54:19.175071 25817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:19.176520 25801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:54:19.176714 25801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/wal
uuid: "fe047c319ae343a19d117e47a25b2b8c"
format_stamp: "Formatted at 2026-06-21 07:54:19 on dist-test-slave-lbpt"
server_key: "bea8d08cfcda134a7c93d21806cc0d73"
server_key_iv: "b421dd1b27f9edaa3e5174752f535859"
server_key_version: "encryptionkey@0"
I20260621 07:54:19.176839 25801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:19.191114 25801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:19.194491 25801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:19.194702 25801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:19.202226 25801 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.60:43037
I20260621 07:54:19.202248 25869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.60:43037 every 8 connection(s)
I20260621 07:54:19.203372 25801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/data/info.pb
I20260621 07:54:19.206208 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 25801
I20260621 07:54:19.206321 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028299065733-12336-0/minicluster-data/master-2/wal/instance
I20260621 07:54:19.206557 25870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:19.206851 12336 external_mini_cluster.cc:1428] Setting key 9482faa6d6f0396056b9f8322ce62759
I20260621 07:54:19.212647 25870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:54:19 dist-test-slave-lbpt krb5kdc[25638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.60@KRBTEST.COM
Jun 21 07:54:19 dist-test-slave-lbpt krb5kdc[25638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028459, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.60@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:19.223706 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.208424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.12.12.60:43037)
0621 07:54:19.208916 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:19.208935 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:19.208981 (+ 46us) server_negotiation.cc:407] Connection header received
0621 07:54:19.210198 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:19.210225 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:19.210754 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:19.211395 (+ 641us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:19.212053 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.213557 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.214655 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.215377 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.215555 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.218732 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:19.218738 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:19.218747 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:19.218773 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:19.220764 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.221255 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.221265 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.221270 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.221389 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.221684 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.221687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.221689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.222075 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:19.222301 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:19.222583 (+ 282us) server_negotiation.cc:299] Negotiation successful
0621 07:54:19.222909 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":137,"threads_started":1}
I20260621 07:54:19.225076 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.214807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:34939 (local address 127.0.0.1:59254)
0621 07:54:19.215342 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.215414 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:19.215575 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:19.215921 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:19.215929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:19.216404 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:19.216538 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:19.216543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.217693 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.217699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:19.218610 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:19.218620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.219111 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.219114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:19.219243 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.219794 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:19.219814 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:19.221479 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:19.223275 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:19.223285 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:19.223291 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:19.223519 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:19.223824 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:19.223827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:19.223829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:19.223933 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:19.224466 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:19.224471 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:19.224590 (+ 119us) client_negotiation.cc:769] Sending connection context
0621 07:54:19.224726 (+ 136us) client_negotiation.cc:240] Negotiation successful
0621 07:54:19.224891 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":199,"threads_started":1}
I20260621 07:54:19.225103 25722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.214841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59254 (local address 127.12.12.62:34939)
0621 07:54:19.215008 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:19.215015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:19.215482 (+ 467us) server_negotiation.cc:407] Connection header received
0621 07:54:19.215675 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:19.215679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:19.215739 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:19.215837 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:19.216706 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.217540 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.218738 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.218972 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.219115 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.221633 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:19.221636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:19.221638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:19.221663 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:19.223153 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.223631 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.223637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.223641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.223724 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.224050 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.224054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.224056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.224315 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:19.224390 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:19.224806 (+ 416us) server_negotiation.cc:299] Negotiation successful
0621 07:54:19.224962 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 07:54:19.226699 25870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:54:19.231803 25719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } attempt: 3
Jun 21 07:54:19 dist-test-slave-lbpt krb5kdc[25638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
Jun 21 07:54:19 dist-test-slave-lbpt krb5kdc[25638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028459, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.60@KRBTEST.COM for kudu/127.12.12.61@KRBTEST.COM
I20260621 07:54:19.235101 25794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } attempt: 3
I20260621 07:54:19.237684 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.227229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40213 (local address 127.0.0.1:54982)
0621 07:54:19.227380 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.227393 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:19.227465 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:19.227788 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:19.227790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:19.227922 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:19.228003 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:19.228008 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.228822 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.228826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:19.229697 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:19.229709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.230256 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.230261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:19.230367 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.231001 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:19.231014 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:19.233205 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:19.235708 (+ 2503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:19.235717 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:19.235719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:19.235947 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:19.236232 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:19.236236 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:19.236238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:19.236342 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:19.237156 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:19.237159 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:19.237246 (+ 87us) client_negotiation.cc:769] Sending connection context
0621 07:54:19.237352 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 07:54:19.237497 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260621 07:54:19.237709 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.227342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54982 (local address 127.12.12.61:40213)
0621 07:54:19.227545 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:19.227550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:19.227564 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:54:19.227609 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:19.227612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:19.227656 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:19.227745 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:19.228123 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.228698 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.229849 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.230124 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.230237 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.233410 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:19.233414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:19.233689 (+ 275us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:19.233728 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:19.235602 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.236053 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.236056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.236058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.236141 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.236452 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.236456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.236458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.236990 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:19.237067 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:19.237412 (+ 345us) server_negotiation.cc:299] Negotiation successful
0621 07:54:19.237545 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:54:19.237706 25722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.226283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59262 (local address 127.12.12.62:34939)
0621 07:54:19.226540 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:19.226544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:19.226574 (+ 30us) server_negotiation.cc:407] Connection header received
0621 07:54:19.226609 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:19.226611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:19.226659 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:19.226743 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:19.227498 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.228191 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.229244 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.229557 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.229639 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.232590 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:19.232596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:19.232657 (+ 61us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:19.232681 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:19.234223 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.234737 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.234741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.234743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.235745 (+ 1002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.236091 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.236095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.236097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.236353 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:19.236617 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:19.237352 (+ 735us) server_negotiation.cc:299] Negotiation successful
0621 07:54:19.237531 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
Jun 21 07:54:19 dist-test-slave-lbpt krb5kdc[25638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028458, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for kudu/127.12.12.60@KRBTEST.COM
I20260621 07:54:19.239792 25870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:54:19 dist-test-slave-lbpt krb5kdc[25638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028458, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.61@KRBTEST.COM for kudu/127.12.12.60@KRBTEST.COM
I20260621 07:54:19.243074 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.232496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.12.12.60:43037)
0621 07:54:19.232627 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:19.232635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:19.232653 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:54:19.232695 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:19.232698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:19.232742 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:19.232836 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:19.233316 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.234234 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.235875 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.236274 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.236394 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.239285 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:19.239289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:19.239294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:19.239325 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:19.241058 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.241547 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.241551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.241553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.241660 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.241977 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.241981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.241983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.242197 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:19.242304 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:19.242769 (+ 465us) server_negotiation.cc:299] Negotiation successful
0621 07:54:19.242898 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:54:19.243613 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.232297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:43037 (local address 127.0.0.1:47482)
0621 07:54:19.232422 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.232437 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:19.232523 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:19.232876 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:19.232881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:19.233047 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:19.233177 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:19.233188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.234385 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.234391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:19.235396 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:19.235410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.236410 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.236416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:19.236512 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.237388 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:19.237402 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:19.239167 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:19.241165 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:19.241168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:19.241170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:19.241426 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:19.241770 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:19.241774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:19.241776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:19.241869 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:19.242360 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:19.242365 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:19.242536 (+ 171us) client_negotiation.cc:769] Sending connection context
0621 07:54:19.243219 (+ 683us) client_negotiation.cc:240] Negotiation successful
0621 07:54:19.243402 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
Jun 21 07:54:19 dist-test-slave-lbpt krb5kdc[25638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.61@KRBTEST.COM
I20260621 07:54:19.247571 25719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187: Bootstrap starting.
I20260621 07:54:19.247732 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.235518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:43037 (local address 127.0.0.1:47484)
0621 07:54:19.235630 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.235646 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:19.235722 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:19.236969 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:19.236973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:19.237153 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:19.237245 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:19.237251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.239345 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.239348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:19.240264 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:19.240273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.240987 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.240994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:19.241109 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.241653 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:19.241667 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:19.243146 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:19.246122 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:19.246125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:19.246127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:19.246380 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:19.246755 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:19.246758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:19.246759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:19.246849 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:19.247271 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:19.247274 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:19.247365 (+ 91us) client_negotiation.cc:769] Sending connection context
0621 07:54:19.247444 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 07:54:19.247579 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260621 07:54:19.248364 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.235580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47484 (local address 127.12.12.60:43037)
0621 07:54:19.236675 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:19.236685 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:19.236697 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:54:19.236730 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:19.236732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:19.236778 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:19.236932 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:19.238004 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.239222 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.240431 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.240843 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.240962 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.243309 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:19.243313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:19.243316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:19.243338 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:19.245983 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.246556 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.246559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.246561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.246648 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.246956 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.246959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.246961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.247159 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:19.247999 (+ 840us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:19.248072 (+ 73us) server_negotiation.cc:299] Negotiation successful
0621 07:54:19.248196 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1011,"thread_start_us":143,"threads_started":1}
I20260621 07:54:19.251027 25719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187: Neither blocks nor log segments found. Creating new log.
Jun 21 07:54:19 dist-test-slave-lbpt krb5kdc[25638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028459, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.60@KRBTEST.COM for kudu/127.12.12.60@KRBTEST.COM
I20260621 07:54:19.251463 25794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba: Bootstrap starting.
I20260621 07:54:19.252056 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.239722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54984 (local address 127.12.12.61:40213)
0621 07:54:19.239839 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:19.239844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:19.239857 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:54:19.239892 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:19.239895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:19.239939 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:19.240030 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:19.241415 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.242267 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.244017 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.244310 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.244398 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.247418 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:19.247423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:19.247425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:19.247458 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:19.249387 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.250327 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.250331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.250334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.250423 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.250843 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.250846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.250848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.251095 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:19.251286 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:19.251735 (+ 449us) server_negotiation.cc:299] Negotiation successful
0621 07:54:19.251870 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:54:19.252452 25719 log.cc:824] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:19.254482 25794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:19.255335 25794 log.cc:824] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:19.255899 25719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187: No bootstrap required, opened a new log
I20260621 07:54:19.256667 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:54:19.257395 25794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba: No bootstrap required, opened a new log
I20260621 07:54:19.257473 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.240295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:43037 (local address 127.0.0.1:47500)
0621 07:54:19.243289 (+ 2994us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:19.243305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:19.243400 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:19.243932 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:19.243935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:19.244418 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:19.244482 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:19.244488 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.247533 (+ 3045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.247536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:19.248606 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:19.248618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.249447 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.249452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:19.249552 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.250059 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:19.250077 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:19.251684 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:19.255150 (+ 3466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:19.255153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:19.255155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:19.256021 (+ 866us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:19.256355 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:19.256357 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:19.256359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:19.256449 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:19.256946 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:19.256949 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:19.257037 (+ 88us) client_negotiation.cc:769] Sending connection context
0621 07:54:19.257133 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 07:54:19.257299 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2926}
I20260621 07:54:19.258154 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.240525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.12.12.60:43037)
0621 07:54:19.241295 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:19.241304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:19.243599 (+ 2295us) server_negotiation.cc:407] Connection header received
0621 07:54:19.243639 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:19.243643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:19.243688 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:19.243772 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:19.246539 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.247415 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.248750 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.249192 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.249307 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.251817 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:19.251824 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:19.251831 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:19.251857 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:19.255024 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.256134 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.256142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.256143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.256253 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.256550 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.256552 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.256554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.256817 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:19.256913 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:19.257784 (+ 871us) server_negotiation.cc:299] Negotiation successful
0621 07:54:19.257906 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":147,"threads_started":1}
I20260621 07:54:19.260502 25719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } }
I20260621 07:54:19.260772 25719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:19.260828 25719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ecb14d607a49a59c9c9754d8a08187, State: Initialized, Role: FOLLOWER
I20260621 07:54:19.261020 25870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c: Bootstrap starting.
I20260621 07:54:19.261301 25719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } }
I20260621 07:54:19.261492 25794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } }
I20260621 07:54:19.261710 25794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:19.261771 25794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41835fabb4cb42f0a5e0b93f130e89ba, State: Initialized, Role: FOLLOWER
I20260621 07:54:19.262023 25883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } } }
I20260621 07:54:19.262254 25883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:54:19.262224 25794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } }
I20260621 07:54:19.262830 25719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:54:19.263489 25794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:54:19.264333 25870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:19.265355 25870 log.cc:824] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c: Log is configured to *not* fsync() on all Append() calls
W20260621 07:54:19.268518 25900 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:54:19.268668 25900 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:54:19.271377 25651 master_runner.cc:186] Error getting master registration for 127.12.12.60:43037: OK, instance_id { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" instance_seqno: 1782028459199982 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 07:54:19.274902 25651 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 07:54:19.275447 25725 master_runner.cc:186] Error getting master registration for 127.12.12.60:43037: OK, instance_id { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" instance_seqno: 1782028459199982 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 07:54:19.279896 25725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 07:54:19.282104 25722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.266769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59268 (local address 127.12.12.62:34939)
0621 07:54:19.266880 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:19.266885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:19.267687 (+ 802us) server_negotiation.cc:407] Connection header received
0621 07:54:19.267886 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:19.267889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:19.267939 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:19.267991 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:19.269070 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.269814 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.271437 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.271766 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.271841 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.277974 (+ 6133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:19.277979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:19.277982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:19.278012 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:19.280176 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.280694 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.280698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.280700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.280787 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.281093 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.281097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.281099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.281325 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:19.281428 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:19.281806 (+ 378us) server_negotiation.cc:299] Negotiation successful
0621 07:54:19.281936 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 07:54:19.282202 25870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c: No bootstrap required, opened a new log
I20260621 07:54:19.283350 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.269061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.12.12.61:40213)
0621 07:54:19.269173 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:19.269178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:19.271364 (+ 2186us) server_negotiation.cc:407] Connection header received
0621 07:54:19.272497 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:19.272501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:19.272549 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:19.272600 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:19.273568 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.274379 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.275821 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.276192 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.276273 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.278208 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:19.278212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:19.278215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:19.278588 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:19.280936 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.281559 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.281563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.281565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.281658 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.282017 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.282021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.282023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.282244 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:19.282346 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:19.283066 (+ 720us) server_negotiation.cc:299] Negotiation successful
0621 07:54:19.283194 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 07:54:19.286569 25884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } } }
I20260621 07:54:19.286718 25884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:54:19.287786 25870 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } }
I20260621 07:54:19.288093 25870 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:19.288162 25870 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe047c319ae343a19d117e47a25b2b8c, State: Initialized, Role: FOLLOWER
I20260621 07:54:19.288724 25870 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } }
I20260621 07:54:19.289265 25913 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } } }
I20260621 07:54:19.289453 25913 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c [sys.catalog]: This master's current role is: FOLLOWER
W20260621 07:54:19.289784 25907 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:54:19.289860 25907 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:54:19.290264 25870 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:54:19.292579 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:19.267967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.12.12.60:43037)
0621 07:54:19.268412 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:19.268417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:19.273752 (+ 5335us) server_negotiation.cc:407] Connection header received
0621 07:54:19.274631 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:19.274634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:19.274685 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:19.274738 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:19.275990 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.276790 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.278354 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:19.278758 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:19.278843 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:19.286179 (+ 7336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:19.286184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:19.286188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:19.286228 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:19.288320 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.290345 (+ 2025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.290349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.290351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.290567 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:19.290941 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:19.290945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:19.290947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:19.291206 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:19.291420 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:19.292229 (+ 809us) server_negotiation.cc:299] Negotiation successful
0621 07:54:19.292399 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348}
W20260621 07:54:19.298026 25924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:54:19.298108 25924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:54:19.300834 25801 master_runner.cc:263] No leader master found from master fe047c319ae343a19d117e47a25b2b8c
I20260621 07:54:19.300905 25801 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 07:54:19.391707 25884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:54:19.391929 25884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } }
I20260621 07:54:19.393090 25884 leader_election.cc:288] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54ecb14d607a49a59c9c9754d8a08187 (127.12.12.62:34939), fe047c319ae343a19d117e47a25b2b8c (127.12.12.60:43037)
I20260621 07:54:19.393884 25693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54ecb14d607a49a59c9c9754d8a08187" is_pre_election: true
I20260621 07:54:19.394254 25693 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41835fabb4cb42f0a5e0b93f130e89ba in term 0.
I20260621 07:54:19.394726 25742 leader_election.cc:302] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41835fabb4cb42f0a5e0b93f130e89ba, 54ecb14d607a49a59c9c9754d8a08187; no voters:
I20260621 07:54:19.395043 25884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:54:19.395123 25884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:54:19.395172 25884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:19.396512 25884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } }
I20260621 07:54:19.397505 25843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe047c319ae343a19d117e47a25b2b8c" is_pre_election: true
I20260621 07:54:19.397840 25843 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41835fabb4cb42f0a5e0b93f130e89ba in term 0.
I20260621 07:54:19.397516 25844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe047c319ae343a19d117e47a25b2b8c"
I20260621 07:54:19.398161 25844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:19.399504 25844 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41835fabb4cb42f0a5e0b93f130e89ba in term 1.
I20260621 07:54:19.400177 25744 leader_election.cc:302] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41835fabb4cb42f0a5e0b93f130e89ba, fe047c319ae343a19d117e47a25b2b8c; no voters:
I20260621 07:54:19.400795 25693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54ecb14d607a49a59c9c9754d8a08187"
I20260621 07:54:19.400954 25693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:19.402264 25693 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41835fabb4cb42f0a5e0b93f130e89ba in term 1.
I20260621 07:54:19.402732 25884 leader_election.cc:288] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [CANDIDATE]: Term 1 election: Requested vote from peers 54ecb14d607a49a59c9c9754d8a08187 (127.12.12.62:34939), fe047c319ae343a19d117e47a25b2b8c (127.12.12.60:43037)
I20260621 07:54:19.402912 25884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:54:19.403021 25884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [term 1 LEADER]: Becoming Leader. State: Replica: 41835fabb4cb42f0a5e0b93f130e89ba, State: Running, Role: LEADER
I20260621 07:54:19.403381 25884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } }
I20260621 07:54:19.408891 25884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41835fabb4cb42f0a5e0b93f130e89ba. Latest consensus state: current_term: 1 leader_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } } }
I20260621 07:54:19.409041 25884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [sys.catalog]: This master's current role is: LEADER
I20260621 07:54:19.409474 25928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:54:19.410537 25928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:54:19.417783 25844 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c [term 1 FOLLOWER]: Refusing update from remote peer 41835fabb4cb42f0a5e0b93f130e89ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:54:19.418499 25926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:54:19.422998 25913 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41835fabb4cb42f0a5e0b93f130e89ba. Latest consensus state: current_term: 1 leader_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } } }
I20260621 07:54:19.423168 25913 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:54:19.426899 25693 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187 [term 1 FOLLOWER]: Refusing update from remote peer 41835fabb4cb42f0a5e0b93f130e89ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:54:19.427531 25926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:54:19.432080 25883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41835fabb4cb42f0a5e0b93f130e89ba. Latest consensus state: current_term: 1 leader_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } } }
I20260621 07:54:19.432220 25883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:54:19.436733 25926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } } }
I20260621 07:54:19.436892 25926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [sys.catalog]: This master's current role is: LEADER
I20260621 07:54:19.438287 25913 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } } }
I20260621 07:54:19.438438 25913 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:54:19.438868 25929 mvcc.cc:205] Tried to move back new op lower bound from 7299188569766584320 to 7299188569730117632. Current Snapshot: MvccSnapshot[applied={T|T < 7299188569766584320}]
I20260621 07:54:19.439257 25930 mvcc.cc:205] Tried to move back new op lower bound from 7299188569766584320 to 7299188569730117632. Current Snapshot: MvccSnapshot[applied={T|T < 7299188569766584320}]
I20260621 07:54:19.445541 25928 catalog_manager.cc:1347] Generated new cluster ID: 2d4bf9b9122f449d83d7fbbe3c9272f8
I20260621 07:54:19.445629 25928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:54:19.451249 25884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } } }
I20260621 07:54:19.451412 25884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba [sys.catalog]: This master's current role is: LEADER
I20260621 07:54:19.452806 25883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ecb14d607a49a59c9c9754d8a08187" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34939 } } peers { permanent_uuid: "41835fabb4cb42f0a5e0b93f130e89ba" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40213 } } peers { permanent_uuid: "fe047c319ae343a19d117e47a25b2b8c" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 43037 } } }
I20260621 07:54:19.452943 25883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:54:19.503690 25928 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:54:19.504901 25928 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:54:19.512516 25928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41835fabb4cb42f0a5e0b93f130e89ba: Generated new TSK 0
I20260621 07:54:19.513391 25928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:54:20.270068 25900 catalog_manager.cc:1259] Loaded cluster ID: 2d4bf9b9122f449d83d7fbbe3c9272f8
I20260621 07:54:20.270157 25900 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187: loading cluster ID for follower catalog manager: success
I20260621 07:54:20.271929 25900 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187: acquiring CA information for follower catalog manager: success
I20260621 07:54:20.273094 25900 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 54ecb14d607a49a59c9c9754d8a08187: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:54:20.299474 25924 catalog_manager.cc:1259] Loaded cluster ID: 2d4bf9b9122f449d83d7fbbe3c9272f8
I20260621 07:54:20.299568 25924 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c: loading cluster ID for follower catalog manager: success
I20260621 07:54:20.301252 25924 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c: acquiring CA information for follower catalog manager: success
I20260621 07:54:20.302451 25924 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fe047c319ae343a19d117e47a25b2b8c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:54:20.581416 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 25651
I20260621 07:54:20.587518 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 25725
I20260621 07:54:20.594054 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 25801
2026-06-21T07:54:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3190 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:54:20 dist-test-slave-lbpt krb5kdc[25944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:54:20 dist-test-slave-lbpt krb5kdc[25944](info): set up 2 sockets
Jun 21 07:54:20 dist-test-slave-lbpt krb5kdc[25944](info): commencing operation
krb5kdc: starting...
W20260621 07:54:21.642865 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:54:21 dist-test-slave-lbpt krb5kdc[25944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:54:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:54:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:54:21.763087 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:42163
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:42163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:54:21.877259 25957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:21.877588 25957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:21.877668 25957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:21.881793 25957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:54:21.881899 25957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:21.881937 25957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:21.881978 25957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:54:21.882014 25957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:54:21.887404 25957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:42163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:42163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:21.888603 25957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:21.889703 25957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:21.898371 25962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:21.898365 25963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:21.898365 25965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:21.898659 25957 server_base.cc:1034] running on GCE node
I20260621 07:54:21.899312 25957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:21.900409 25957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:21.901600 25957 hybrid_clock.cc:648] HybridClock initialized: now 1782028461901564 us; error 51 us; skew 500 ppm
Jun 21 07:54:21 dist-test-slave-lbpt krb5kdc[25944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028461, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:21.904594 25957 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:54:21.905529 25957 webserver.cc:460] Webserver started at http://127.12.12.62:37941/ using document root <none> and password file <none>
I20260621 07:54:21.906126 25957 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:21.906234 25957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:21.906528 25957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:54:21.908605 25957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "476661e0fe6f48b18047cb3a9923db0c"
format_stamp: "Formatted at 2026-06-21 07:54:21 on dist-test-slave-lbpt"
server_key: "64686d3f299ffb407869f640b7010eb9"
server_key_iv: "ac0849f339479346773dc3f6bed70a66"
server_key_version: "encryptionkey@0"
I20260621 07:54:21.909197 25957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "476661e0fe6f48b18047cb3a9923db0c"
format_stamp: "Formatted at 2026-06-21 07:54:21 on dist-test-slave-lbpt"
server_key: "64686d3f299ffb407869f640b7010eb9"
server_key_iv: "ac0849f339479346773dc3f6bed70a66"
server_key_version: "encryptionkey@0"
I20260621 07:54:21.913115 25957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:54:21.915772 25972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:21.917122 25957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:54:21.917273 25957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "476661e0fe6f48b18047cb3a9923db0c"
format_stamp: "Formatted at 2026-06-21 07:54:21 on dist-test-slave-lbpt"
server_key: "64686d3f299ffb407869f640b7010eb9"
server_key_iv: "ac0849f339479346773dc3f6bed70a66"
server_key_version: "encryptionkey@0"
I20260621 07:54:21.917399 25957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:21.933413 25957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:21.936481 25957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:21.936705 25957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:21.944271 25957 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:42163
I20260621 07:54:21.944288 26024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:42163 every 8 connection(s)
I20260621 07:54:21.945258 25957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:54:21.948207 26025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:21.949673 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 25957
I20260621 07:54:21.949802 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:54:21.950115 12336 external_mini_cluster.cc:1428] Setting key 4e42471503b5d16a5243dc6a9d2b2493
I20260621 07:54:21.957669 26025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c: Bootstrap starting.
I20260621 07:54:21.961040 26025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:21.962253 26025 log.cc:824] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:54:21 dist-test-slave-lbpt krb5kdc[25944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:21.965536 26025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c: No bootstrap required, opened a new log
I20260621 07:54:21.970973 26025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476661e0fe6f48b18047cb3a9923db0c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42163 } }
I20260621 07:54:21.971271 26025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:21.971330 26025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476661e0fe6f48b18047cb3a9923db0c, State: Initialized, Role: FOLLOWER
I20260621 07:54:21.971889 26025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476661e0fe6f48b18047cb3a9923db0c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42163 } }
I20260621 07:54:21.972045 26025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:54:21.972118 26025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:54:21.972210 26025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:21.972404 26028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:21.951578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.12.12.62:42163)
0621 07:54:21.952485 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:21.952506 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:21.952549 (+ 43us) server_negotiation.cc:407] Connection header received
0621 07:54:21.953449 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:21.953478 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:21.953975 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:21.954402 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:21.956187 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:21.957877 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:21.959366 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:21.960295 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:21.960458 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:21.965900 (+ 5442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:21.965907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:21.965933 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:21.965972 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:21.968292 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:21.968925 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:21.968931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:21.968938 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:21.969087 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:21.969405 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:21.969408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:21.969410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:21.969958 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:21.970156 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:21.971028 (+ 872us) server_negotiation.cc:299] Negotiation successful
0621 07:54:21.971390 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":126,"threads_started":1}
I20260621 07:54:21.973403 26025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476661e0fe6f48b18047cb3a9923db0c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42163 } }
I20260621 07:54:21.974890 26025 leader_election.cc:302] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 476661e0fe6f48b18047cb3a9923db0c; no voters:
I20260621 07:54:21.975581 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:54:21.975728 26025 leader_election.cc:288] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:54:21.975790 26030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:54:21.976025 26030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [term 1 LEADER]: Becoming Leader. State: Replica: 476661e0fe6f48b18047cb3a9923db0c, State: Running, Role: LEADER
I20260621 07:54:21.976387 26030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476661e0fe6f48b18047cb3a9923db0c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42163 } }
I20260621 07:54:21.978179 26031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 476661e0fe6f48b18047cb3a9923db0c. Latest consensus state: current_term: 1 leader_uuid: "476661e0fe6f48b18047cb3a9923db0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476661e0fe6f48b18047cb3a9923db0c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42163 } } }
I20260621 07:54:21.978307 26031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [sys.catalog]: This master's current role is: LEADER
I20260621 07:54:21.978544 26031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "476661e0fe6f48b18047cb3a9923db0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476661e0fe6f48b18047cb3a9923db0c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 42163 } } }
I20260621 07:54:21.978618 26031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [sys.catalog]: This master's current role is: LEADER
I20260621 07:54:21.978988 26025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:54:21.980365 26033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:54:21.984328 26033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:54:21.993435 26033 catalog_manager.cc:1347] Generated new cluster ID: 67b9e921e01843ada6833f28d6fa9ae0
I20260621 07:54:21.993566 26033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:54:22.021090 26033 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:54:22.022616 26033 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:54:22.038666 26033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 476661e0fe6f48b18047cb3a9923db0c: Generated new TSK 0
I20260621 07:54:22.039615 26033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:54:22.082633 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 25957
2026-06-21T07:54:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1489 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:54:22 dist-test-slave-lbpt krb5kdc[26051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:54:22 dist-test-slave-lbpt krb5kdc[26051](info): set up 2 sockets
Jun 21 07:54:22 dist-test-slave-lbpt krb5kdc[26051](info): commencing operation
krb5kdc: starting...
W20260621 07:54:23.142674 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:54:23 dist-test-slave-lbpt krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:54:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:54:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:54:23.264999 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:46527
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:46527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:54:23.378317 26064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:23.378707 26064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:23.378752 26064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:23.382803 26064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:54:23.382867 26064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:23.382887 26064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:23.382905 26064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:54:23.382920 26064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:54:23.388407 26064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:46527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:46527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:23.389506 26064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:23.390614 26064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:23.399447 26072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:23.399523 26070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:23.399681 26064 server_base.cc:1034] running on GCE node
W20260621 07:54:23.399447 26069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:23.400262 26064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:23.401398 26064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:23.402616 26064 hybrid_clock.cc:648] HybridClock initialized: now 1782028463402564 us; error 67 us; skew 500 ppm
Jun 21 07:54:23 dist-test-slave-lbpt krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028463, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:23.405915 26064 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:54:23.407135 26064 webserver.cc:460] Webserver started at http://127.12.12.62:44553/ using document root <none> and password file <none>
I20260621 07:54:23.407757 26064 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:23.407858 26064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:23.408116 26064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:54:23.410305 26064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "b8b691e0131b432db6295d8af180bc09"
format_stamp: "Formatted at 2026-06-21 07:54:23 on dist-test-slave-lbpt"
server_key: "b1be7e0495772f8ae905a86eface7779"
server_key_iv: "9fd5ff728564a830bb3d38370862edaa"
server_key_version: "encryptionkey@0"
I20260621 07:54:23.410983 26064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "b8b691e0131b432db6295d8af180bc09"
format_stamp: "Formatted at 2026-06-21 07:54:23 on dist-test-slave-lbpt"
server_key: "b1be7e0495772f8ae905a86eface7779"
server_key_iv: "9fd5ff728564a830bb3d38370862edaa"
server_key_version: "encryptionkey@0"
I20260621 07:54:23.415022 26064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:54:23.417564 26079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:23.418921 26064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:54:23.419080 26064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "b8b691e0131b432db6295d8af180bc09"
format_stamp: "Formatted at 2026-06-21 07:54:23 on dist-test-slave-lbpt"
server_key: "b1be7e0495772f8ae905a86eface7779"
server_key_iv: "9fd5ff728564a830bb3d38370862edaa"
server_key_version: "encryptionkey@0"
I20260621 07:54:23.419202 26064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:23.438575 26064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:23.441614 26064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:23.441833 26064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:23.448962 26064 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:46527
I20260621 07:54:23.449254 26131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:46527 every 8 connection(s)
I20260621 07:54:23.449971 26064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:54:23.451735 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 26064
I20260621 07:54:23.451861 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:54:23.452147 12336 external_mini_cluster.cc:1428] Setting key 9b94542ebf5d05a0c32f8244d0e45d53
I20260621 07:54:23.453073 26132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:23.459864 26132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09: Bootstrap starting.
Jun 21 07:54:23 dist-test-slave-lbpt krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:23.462591 26132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:23.463569 26132 log.cc:824] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:23.466055 26132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09: No bootstrap required, opened a new log
I20260621 07:54:23.467825 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:23.453531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.12.12.62:46527)
0621 07:54:23.453944 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:23.453962 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:23.454005 (+ 43us) server_negotiation.cc:407] Connection header received
0621 07:54:23.454887 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:23.454912 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:23.455333 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:23.455696 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:23.456357 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:23.457836 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:23.458841 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:23.459519 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:23.459679 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:23.462546 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:23.462553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:23.462571 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:23.462611 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:23.464635 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:23.465060 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:23.465065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:23.465072 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:23.465225 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:23.465482 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:23.465485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:23.465487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:23.466006 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:23.466220 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:23.466536 (+ 316us) server_negotiation.cc:299] Negotiation successful
0621 07:54:23.466847 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
I20260621 07:54:23.470654 26132 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } }
I20260621 07:54:23.470898 26132 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:23.470944 26132 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b691e0131b432db6295d8af180bc09, State: Initialized, Role: FOLLOWER
I20260621 07:54:23.471379 26132 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } }
I20260621 07:54:23.471506 26132 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:54:23.471549 26132 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:54:23.471630 26132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:23.472568 26132 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } }
I20260621 07:54:23.472882 26132 leader_election.cc:302] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8b691e0131b432db6295d8af180bc09; no voters:
I20260621 07:54:23.473171 26132 leader_election.cc:288] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:54:23.473295 26137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:54:23.473528 26137 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 1 LEADER]: Becoming Leader. State: Replica: b8b691e0131b432db6295d8af180bc09, State: Running, Role: LEADER
I20260621 07:54:23.473856 26137 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } }
I20260621 07:54:23.474586 26132 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:54:23.475181 26138 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b691e0131b432db6295d8af180bc09. Latest consensus state: current_term: 1 leader_uuid: "b8b691e0131b432db6295d8af180bc09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } } }
I20260621 07:54:23.475368 26138 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [sys.catalog]: This master's current role is: LEADER
I20260621 07:54:23.475644 26139 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8b691e0131b432db6295d8af180bc09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } } }
I20260621 07:54:23.475765 26139 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [sys.catalog]: This master's current role is: LEADER
I20260621 07:54:23.475818 26144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:54:23.478708 26144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:54:23.484479 26144 catalog_manager.cc:1347] Generated new cluster ID: 8f61df329bd7409fa10b65bc5f2ae66b
I20260621 07:54:23.484561 26144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:54:23.506875 26144 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:54:23.507982 26144 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:54:23.519946 26144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09: Generated new TSK 0
I20260621 07:54:23.520884 26144 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:54:23.550302 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:46527
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:54:23.667403 26158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:23.667719 26158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:23.667804 26158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:23.672047 26158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:23.672130 26158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:23.672195 26158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:54:23.678185 26158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:46527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:23.679402 26158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:23.680516 26158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:23.691350 26164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:23.691386 26163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:23.691462 26158 server_base.cc:1034] running on GCE node
W20260621 07:54:23.691359 26166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:23.692111 26158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:23.692759 26158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:23.693974 26158 hybrid_clock.cc:648] HybridClock initialized: now 1782028463693938 us; error 51 us; skew 500 ppm
Jun 21 07:54:23 dist-test-slave-lbpt krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028463, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:23.697170 26158 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:54:23.698169 26158 webserver.cc:460] Webserver started at http://127.12.12.1:36367/ using document root <none> and password file <none>
I20260621 07:54:23.698853 26158 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:23.698951 26158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:23.699184 26158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:54:23.701256 26158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "435bf73c81a140d885d3bd500094f182"
format_stamp: "Formatted at 2026-06-21 07:54:23 on dist-test-slave-lbpt"
server_key: "952c344175f7f5645a1fcf26bcf58682"
server_key_iv: "f2a7e03ae8be226d7ec9b7f09e2a3bc6"
server_key_version: "encryptionkey@0"
I20260621 07:54:23.701835 26158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "435bf73c81a140d885d3bd500094f182"
format_stamp: "Formatted at 2026-06-21 07:54:23 on dist-test-slave-lbpt"
server_key: "952c344175f7f5645a1fcf26bcf58682"
server_key_iv: "f2a7e03ae8be226d7ec9b7f09e2a3bc6"
server_key_version: "encryptionkey@0"
I20260621 07:54:23.705720 26158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 07:54:23.708459 26173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:23.709949 26158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:54:23.710106 26158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "435bf73c81a140d885d3bd500094f182"
format_stamp: "Formatted at 2026-06-21 07:54:23 on dist-test-slave-lbpt"
server_key: "952c344175f7f5645a1fcf26bcf58682"
server_key_iv: "f2a7e03ae8be226d7ec9b7f09e2a3bc6"
server_key_version: "encryptionkey@0"
I20260621 07:54:23.710246 26158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:23.725474 26158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:23.728765 26158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:23.728991 26158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:23.729621 26158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:23.730562 26158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:54:23.730646 26158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:23.730736 26158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:54:23.730785 26158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:23.739864 26158 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:45081
I20260621 07:54:23.739879 26286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:45081 every 8 connection(s)
I20260621 07:54:23.740836 26158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:54:23.747417 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 26158
I20260621 07:54:23.747599 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:54:23.747911 12336 external_mini_cluster.cc:1428] Setting key bf061e6b5fdddf4e7035e50c96dfaca8
Jun 21 07:54:23 dist-test-slave-lbpt krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028463, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:23.756978 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:23.742857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:37155 (local address 127.12.12.62:46527)
0621 07:54:23.743000 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:23.743004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:23.743866 (+ 862us) server_negotiation.cc:407] Connection header received
0621 07:54:23.745387 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:23.745391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:23.745450 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:23.745541 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:23.746794 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:23.747369 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:23.748811 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:23.749058 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:23.749182 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:23.752498 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:23.752502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:23.752504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:23.752531 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:23.754677 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:23.755286 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:23.755290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:23.755292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:23.755378 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:23.755809 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:23.755814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:23.755815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:23.756019 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:23.756130 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:23.756664 (+ 534us) server_negotiation.cc:299] Negotiation successful
0621 07:54:23.756807 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
I20260621 07:54:23.757998 26289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:23.743205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:37155)
0621 07:54:23.743692 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:23.743739 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:23.745102 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:23.745743 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:23.745750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:23.746118 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:23.746636 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:23.746651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:23.747508 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:23.747517 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:23.748686 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:23.748697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:23.749226 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:23.749231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:23.749464 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:23.750190 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:23.750218 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:23.752291 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:23.754818 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:23.754824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:23.754843 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:23.755160 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:23.755483 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:23.755489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:23.755493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:23.755704 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:23.756134 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:23.756144 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:23.756418 (+ 274us) client_negotiation.cc:769] Sending connection context
0621 07:54:23.756627 (+ 209us) client_negotiation.cc:240] Negotiation successful
0621 07:54:23.757001 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":107,"threads_started":1}
I20260621 07:54:23.759434 26287 heartbeater.cc:347] Connected to a master server at 127.12.12.62:46527
I20260621 07:54:23.759752 26287 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
I20260621 07:54:23.760331 26287 heartbeater.cc:511] Master 127.12.12.62:46527 requested a full tablet report, sending...
I20260621 07:54:23.762055 26096 ts_manager.cc:194] Registered new tserver with Master: 435bf73c81a140d885d3bd500094f182 (127.12.12.1:45081)
I20260621 07:54:23.763263 26096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:37155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:54:23.778784 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:46527
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:54:23.890373 26292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:23.890772 26292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:23.890820 26292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:23.894979 26292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:23.895046 26292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:23.895108 26292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:54:23.900507 26292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:46527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:23.901595 26292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:23.902716 26292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:54:23.912585 26292 server_base.cc:1034] running on GCE node
W20260621 07:54:23.912597 26300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:23.912598 26297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:23.912598 26298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:23.913352 26292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:23.914027 26292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:23.915225 26292 hybrid_clock.cc:648] HybridClock initialized: now 1782028463915186 us; error 50 us; skew 500 ppm
Jun 21 07:54:23 dist-test-slave-lbpt krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028463, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:23.918015 26292 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:54:23.919011 26292 webserver.cc:460] Webserver started at http://127.12.12.2:33763/ using document root <none> and password file <none>
I20260621 07:54:23.919612 26292 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:23.919705 26292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:23.919961 26292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:54:23.922012 26292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "d5092a43d05643658a7da11156b33da9"
format_stamp: "Formatted at 2026-06-21 07:54:23 on dist-test-slave-lbpt"
server_key: "82a4c47d4c9c0a25f7afa5889f762a99"
server_key_iv: "ed4e0725d31c950ea170a13c117bf923"
server_key_version: "encryptionkey@0"
I20260621 07:54:23.922704 26292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "d5092a43d05643658a7da11156b33da9"
format_stamp: "Formatted at 2026-06-21 07:54:23 on dist-test-slave-lbpt"
server_key: "82a4c47d4c9c0a25f7afa5889f762a99"
server_key_iv: "ed4e0725d31c950ea170a13c117bf923"
server_key_version: "encryptionkey@0"
I20260621 07:54:23.926785 26292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 07:54:23.929283 26307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:23.930560 26292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:54:23.930692 26292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "d5092a43d05643658a7da11156b33da9"
format_stamp: "Formatted at 2026-06-21 07:54:23 on dist-test-slave-lbpt"
server_key: "82a4c47d4c9c0a25f7afa5889f762a99"
server_key_iv: "ed4e0725d31c950ea170a13c117bf923"
server_key_version: "encryptionkey@0"
I20260621 07:54:23.930804 26292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:23.950950 26292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:23.954111 26292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:23.954331 26292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:23.954988 26292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:23.955844 26292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:54:23.955914 26292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:23.955986 26292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:54:23.956037 26292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:23.965209 26292 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:38033
I20260621 07:54:23.965222 26420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:38033 every 8 connection(s)
I20260621 07:54:23.966204 26292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:54:23.975780 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 26292
I20260621 07:54:23.975908 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:54:23.976243 12336 external_mini_cluster.cc:1428] Setting key a88eee5766b6200fdd858fa2b55c00b3
Jun 21 07:54:23 dist-test-slave-lbpt krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028463, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:23.981577 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:23.968306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:43081 (local address 127.12.12.62:46527)
0621 07:54:23.968464 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:23.968469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:23.969386 (+ 917us) server_negotiation.cc:407] Connection header received
0621 07:54:23.970790 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:23.970793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:23.970843 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:23.970911 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:23.972143 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:23.972667 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:23.973893 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:23.974123 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:23.974264 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:23.977492 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:23.977495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:23.977497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:23.977524 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:23.979351 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:23.979952 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:23.979956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:23.979958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:23.980045 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:23.980409 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:23.980413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:23.980414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:23.980603 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:23.980710 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:23.981338 (+ 628us) server_negotiation.cc:299] Negotiation successful
0621 07:54:23.981443 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 07:54:23.982909 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:23.968755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:43081)
0621 07:54:23.969222 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:23.969268 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:23.970564 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:23.971083 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:23.971090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:23.971483 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:23.971985 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:23.971999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:23.972804 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:23.972808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:23.973768 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:23.973778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:23.974323 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:23.974328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:23.974633 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:23.975295 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:23.975333 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:23.977300 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:23.979483 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:23.979490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:23.979507 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:23.979834 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:23.980142 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:23.980146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:23.980148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:23.980302 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:23.980716 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:23.980725 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:23.981060 (+ 335us) client_negotiation.cc:769] Sending connection context
0621 07:54:23.981282 (+ 222us) client_negotiation.cc:240] Negotiation successful
0621 07:54:23.981664 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":111,"threads_started":1}
I20260621 07:54:23.984259 26421 heartbeater.cc:347] Connected to a master server at 127.12.12.62:46527
I20260621 07:54:23.984535 26421 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:23.985076 26421 heartbeater.cc:511] Master 127.12.12.62:46527 requested a full tablet report, sending...
I20260621 07:54:23.986177 26096 ts_manager.cc:194] Registered new tserver with Master: d5092a43d05643658a7da11156b33da9 (127.12.12.2:38033)
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
I20260621 07:54:23.986801 26096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:43081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:54:24.010164 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:46527
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:54:24.127388 26426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:24.127674 26426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:24.127720 26426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:24.131927 26426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:24.132001 26426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:24.132063 26426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:54:24.137748 26426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:46527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:24.138825 26426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:24.139818 26426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:24.149346 26434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:24.149351 26432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:24.149402 26431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:24.149706 26426 server_base.cc:1034] running on GCE node
I20260621 07:54:24.150120 26426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:24.150816 26426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:24.152010 26426 hybrid_clock.cc:648] HybridClock initialized: now 1782028464151992 us; error 29 us; skew 500 ppm
Jun 21 07:54:24 dist-test-slave-lbpt krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028464, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:24.154950 26426 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:54:24.155777 26426 webserver.cc:460] Webserver started at http://127.12.12.3:38413/ using document root <none> and password file <none>
I20260621 07:54:24.156280 26426 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:24.156337 26426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:24.156492 26426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:54:24.158528 26426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "d01453ed24c64788a8db8af1b1e27947"
format_stamp: "Formatted at 2026-06-21 07:54:24 on dist-test-slave-lbpt"
server_key: "dac034e32b1454a52680901fe4ac823d"
server_key_iv: "e1af86d3c2ef7e2b7f88b009892a81c0"
server_key_version: "encryptionkey@0"
I20260621 07:54:24.159046 26426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "d01453ed24c64788a8db8af1b1e27947"
format_stamp: "Formatted at 2026-06-21 07:54:24 on dist-test-slave-lbpt"
server_key: "dac034e32b1454a52680901fe4ac823d"
server_key_iv: "e1af86d3c2ef7e2b7f88b009892a81c0"
server_key_version: "encryptionkey@0"
I20260621 07:54:24.162701 26426 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260621 07:54:24.165174 26441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:24.166716 26426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:54:24.166841 26426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "d01453ed24c64788a8db8af1b1e27947"
format_stamp: "Formatted at 2026-06-21 07:54:24 on dist-test-slave-lbpt"
server_key: "dac034e32b1454a52680901fe4ac823d"
server_key_iv: "e1af86d3c2ef7e2b7f88b009892a81c0"
server_key_version: "encryptionkey@0"
I20260621 07:54:24.166954 26426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:24.180703 26426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:24.184286 26426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:24.184501 26426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:24.185140 26426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:24.185971 26426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:54:24.186015 26426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:24.186090 26426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:54:24.186129 26426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:24.195448 26426 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:36083
I20260621 07:54:24.195467 26554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:36083 every 8 connection(s)
I20260621 07:54:24.196427 26426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:54:24.197736 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 26426
I20260621 07:54:24.197853 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:54:24.198171 12336 external_mini_cluster.cc:1428] Setting key f0ea1ec9013e7e8f0caaba35ce86a817
Jun 21 07:54:24 dist-test-slave-lbpt krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028464, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:24.212015 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.198646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:44723 (local address 127.12.12.62:46527)
0621 07:54:24.198809 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:24.198814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:24.199978 (+ 1164us) server_negotiation.cc:407] Connection header received
0621 07:54:24.201459 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:24.201462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:24.201509 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:24.201579 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:24.202840 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.203379 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.204562 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.204785 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.205023 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.207637 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:24.207640 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:24.207642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:24.207669 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:24.209360 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:24.210008 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:24.210011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:24.210013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:24.210097 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:24.210557 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:24.210560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:24.210561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:24.210807 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:24.210898 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:24.211675 (+ 777us) server_negotiation.cc:299] Negotiation successful
0621 07:54:24.211820 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:54:24.212951 26557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.199168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:44723)
0621 07:54:24.199784 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:24.199864 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:24.201251 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:24.201739 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:24.201746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:24.202117 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:24.202696 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:24.202712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.203500 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.203503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:24.204447 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:24.204457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.205054 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.205056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:24.205209 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.205778 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:24.205793 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:24.207417 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:24.209525 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:24.209531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:24.209545 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:24.209862 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:24.210203 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:24.210206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:24.210208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:24.210361 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:24.210921 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:24.210927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:24.211247 (+ 320us) client_negotiation.cc:769] Sending connection context
0621 07:54:24.211475 (+ 228us) client_negotiation.cc:240] Negotiation successful
0621 07:54:24.211837 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":136,"threads_started":1}
I20260621 07:54:24.214069 26555 heartbeater.cc:347] Connected to a master server at 127.12.12.62:46527
I20260621 07:54:24.214524 26555 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:24.215052 26555 heartbeater.cc:511] Master 127.12.12.62:46527 requested a full tablet report, sending...
I20260621 07:54:24.216038 26096 ts_manager.cc:194] Registered new tserver with Master: d01453ed24c64788a8db8af1b1e27947 (127.12.12.3:36083)
I20260621 07:54:24.216638 26096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:44723
I20260621 07:54:24.224435 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:54:24.233829 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.226378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.12.12.62:46527)
0621 07:54:24.226570 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:24.226575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:24.226752 (+ 177us) server_negotiation.cc:407] Connection header received
0621 07:54:24.226905 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:24.226909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:24.226961 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:24.227046 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:24.227758 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.228279 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.229086 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.229309 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.229413 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.230810 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:24.230814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:24.230816 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:24.230845 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:24.232112 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:24.232535 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:24.232540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:24.232542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:24.232627 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:24.232912 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:24.232917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:24.232919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:24.233153 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:24.233299 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:24.233529 (+ 230us) server_negotiation.cc:299] Negotiation successful
0621 07:54:24.233646 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260621 07:54:24.236969 26096 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:54:24.240145 26096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:54:24.254758 26566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.248716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:38033 (local address 127.0.0.1:33498)
0621 07:54:24.249603 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:24.249617 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:24.249740 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:24.250547 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:24.250550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:24.250568 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:24.250674 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:24.250681 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.251740 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.251744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:24.253163 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:24.253172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.254202 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.254205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:24.254311 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.254350 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 07:54:24.254478 (+ 128us) client_negotiation.cc:240] Negotiation successful
0621 07:54:24.254591 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":41,"thread_start_us":124,"threads_started":1}
I20260621 07:54:24.255007 26571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.249684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51386 (local address 127.12.12.1:45081)
0621 07:54:24.250021 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:24.250033 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:24.250372 (+ 339us) server_negotiation.cc:407] Connection header received
0621 07:54:24.250508 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:24.250516 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:24.250716 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:24.250855 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:24.251033 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.252101 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.253490 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.254297 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.254398 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.254552 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:24.254745 (+ 193us) server_negotiation.cc:299] Negotiation successful
0621 07:54:24.254847 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":71,"threads_started":1}
I20260621 07:54:24.255358 26569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.249279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:45081 (local address 127.0.0.1:51386)
0621 07:54:24.250132 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:24.250142 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:24.250264 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:24.250840 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:24.250843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:24.250860 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:24.250939 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:24.250945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.252235 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.252238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:24.253367 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:24.253375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.254487 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.254491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:24.254581 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.254603 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:54:24.254696 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 07:54:24.254772 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":31,"thread_start_us":81,"threads_started":1}
I20260621 07:54:24.255378 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.249319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.12.12.2:38033)
0621 07:54:24.250039 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:24.250049 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:24.250070 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:54:24.250137 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:24.250142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:24.250321 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:24.250568 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:24.250775 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.251593 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.253282 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.254051 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.254212 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.254367 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:24.255144 (+ 777us) server_negotiation.cc:299] Negotiation successful
0621 07:54:24.255238 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":108,"threads_started":1}
I20260621 07:54:24.255681 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.248551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:36083 (local address 127.0.0.1:46216)
0621 07:54:24.249096 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:24.249147 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:24.249272 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:24.250272 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:24.250278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:24.250301 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:24.250465 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:24.250472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.252078 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.252081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:24.254017 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:24.254025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.255299 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.255303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:24.255395 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.255423 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 07:54:24.255498 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 07:54:24.255562 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":19,"thread_start_us":71,"threads_started":1}
I20260621 07:54:24.255980 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.248722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46216 (local address 127.12.12.3:36083)
0621 07:54:24.249762 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:24.249777 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:24.249802 (+ 25us) server_negotiation.cc:407] Connection header received
0621 07:54:24.249881 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:24.249888 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:24.250103 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:24.250268 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:24.250612 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.251840 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.254147 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.254993 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.255096 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.255239 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:24.255765 (+ 526us) server_negotiation.cc:299] Negotiation successful
0621 07:54:24.255853 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":936,"thread_start_us":153,"threads_started":1}
I20260621 07:54:24.258461 26489 tablet_service.cc:1467] Processing CreateTablet for tablet c06c05a45c23436c8380fd8ab910d0c8 (DEFAULT_TABLE table=test-table [id=b44d7ab11272422cab0d6a704d0668dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:54:24.258469 26221 tablet_service.cc:1467] Processing CreateTablet for tablet c06c05a45c23436c8380fd8ab910d0c8 (DEFAULT_TABLE table=test-table [id=b44d7ab11272422cab0d6a704d0668dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:54:24.258455 26355 tablet_service.cc:1467] Processing CreateTablet for tablet c06c05a45c23436c8380fd8ab910d0c8 (DEFAULT_TABLE table=test-table [id=b44d7ab11272422cab0d6a704d0668dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:54:24.259500 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06c05a45c23436c8380fd8ab910d0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:24.259513 26221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06c05a45c23436c8380fd8ab910d0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:24.259513 26355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06c05a45c23436c8380fd8ab910d0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:24.265609 26572 tablet_bootstrap.cc:492] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947: Bootstrap starting.
I20260621 07:54:24.266901 26573 tablet_bootstrap.cc:492] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182: Bootstrap starting.
I20260621 07:54:24.268317 26572 tablet_bootstrap.cc:654] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:24.268745 26573 tablet_bootstrap.cc:654] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:24.269085 26572 log.cc:824] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:24.269462 26573 log.cc:824] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:24.269761 26574 tablet_bootstrap.cc:492] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9: Bootstrap starting.
I20260621 07:54:24.271226 26572 tablet_bootstrap.cc:492] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947: No bootstrap required, opened a new log
I20260621 07:54:24.271418 26572 ts_tablet_manager.cc:1397] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260621 07:54:24.271500 26573 tablet_bootstrap.cc:492] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182: No bootstrap required, opened a new log
I20260621 07:54:24.271684 26573 ts_tablet_manager.cc:1397] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:54:24.272185 26574 tablet_bootstrap.cc:654] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:24.273017 26574 log.cc:824] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:24.274849 26574 tablet_bootstrap.cc:492] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9: No bootstrap required, opened a new log
I20260621 07:54:24.275051 26574 ts_tablet_manager.cc:1397] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:54:24.275997 26573 raft_consensus.cc:348] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:24.275996 26572 raft_consensus.cc:348] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:24.276314 26573 raft_consensus.cc:374] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:24.276335 26572 raft_consensus.cc:374] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:24.276371 26573 raft_consensus.cc:729] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435bf73c81a140d885d3bd500094f182, State: Initialized, Role: FOLLOWER
I20260621 07:54:24.276384 26572 raft_consensus.cc:729] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d01453ed24c64788a8db8af1b1e27947, State: Initialized, Role: FOLLOWER
I20260621 07:54:24.276889 26573 consensus_queue.cc:260] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:24.276889 26572 consensus_queue.cc:260] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:24.277606 26555 heartbeater.cc:503] Master 127.12.12.62:46527 was elected leader, sending a full tablet report...
I20260621 07:54:24.277940 26573 ts_tablet_manager.cc:1428] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260621 07:54:24.277940 26572 ts_tablet_manager.cc:1428] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260621 07:54:24.278272 26287 heartbeater.cc:503] Master 127.12.12.62:46527 was elected leader, sending a full tablet report...
I20260621 07:54:24.279644 26574 raft_consensus.cc:348] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:24.279915 26574 raft_consensus.cc:374] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:24.279973 26574 raft_consensus.cc:729] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5092a43d05643658a7da11156b33da9, State: Initialized, Role: FOLLOWER
I20260621 07:54:24.280521 26574 consensus_queue.cc:260] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:24.281562 26574 ts_tablet_manager.cc:1428] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:54:24.281929 26421 heartbeater.cc:503] Master 127.12.12.62:46527 was elected leader, sending a full tablet report...
I20260621 07:54:24.399420 26578 raft_consensus.cc:482] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:54:24.399672 26578 raft_consensus.cc:504] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:24.400895 26578 leader_election.cc:288] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d01453ed24c64788a8db8af1b1e27947 (127.12.12.3:36083), d5092a43d05643658a7da11156b33da9 (127.12.12.2:38033)
I20260621 07:54:24.406675 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.401423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:41063 (local address 127.12.12.2:38033)
0621 07:54:24.401561 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:24.401566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:24.401874 (+ 308us) server_negotiation.cc:407] Connection header received
0621 07:54:24.402050 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:24.402055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:24.402113 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:24.402190 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:24.402517 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.403384 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.405313 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.405951 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.406065 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.406132 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:24.406454 (+ 322us) server_negotiation.cc:299] Negotiation successful
0621 07:54:24.406540 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 07:54:24.406675 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.401331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:40345 (local address 127.12.12.3:36083)
0621 07:54:24.401478 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:24.401484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:24.402013 (+ 529us) server_negotiation.cc:407] Connection header received
0621 07:54:24.402216 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:24.402219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:24.402271 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:24.402335 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:24.402651 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.403379 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.405313 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.405946 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.406065 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.406138 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:24.406458 (+ 320us) server_negotiation.cc:299] Negotiation successful
0621 07:54:24.406540 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:54:24.406725 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.401190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:36083 (local address 127.12.12.1:40345)
0621 07:54:24.401894 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:24.401908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:24.402083 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:24.402351 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:24.402354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:24.402366 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:24.402538 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:24.402543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.403536 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.403540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:24.405157 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:24.405169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.406082 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.406086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:24.406250 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.406321 (+ 71us) client_negotiation.cc:769] Sending connection context
0621 07:54:24.406401 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 07:54:24.406532 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":584,"thread_start_us":99,"threads_started":1}
I20260621 07:54:24.406919 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.401190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:38033 (local address 127.12.12.1:41063)
0621 07:54:24.401729 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:24.401766 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:24.401914 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:24.402221 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:24.402223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:24.402236 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:24.402359 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:24.402366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.403536 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.403540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:24.405162 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:24.405169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.406082 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.406086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:24.406279 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.406329 (+ 50us) client_negotiation.cc:769] Sending connection context
0621 07:54:24.406401 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 07:54:24.406532 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":230,"thread_start_us":145,"threads_started":1}
I20260621 07:54:24.407546 26375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06c05a45c23436c8380fd8ab910d0c8" candidate_uuid: "435bf73c81a140d885d3bd500094f182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5092a43d05643658a7da11156b33da9" is_pre_election: true
I20260621 07:54:24.407543 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06c05a45c23436c8380fd8ab910d0c8" candidate_uuid: "435bf73c81a140d885d3bd500094f182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d01453ed24c64788a8db8af1b1e27947" is_pre_election: true
I20260621 07:54:24.407861 26509 raft_consensus.cc:2454] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 435bf73c81a140d885d3bd500094f182 in term 0.
I20260621 07:54:24.407862 26375 raft_consensus.cc:2454] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 435bf73c81a140d885d3bd500094f182 in term 0.
I20260621 07:54:24.408360 26175 leader_election.cc:302] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 435bf73c81a140d885d3bd500094f182, d01453ed24c64788a8db8af1b1e27947, d5092a43d05643658a7da11156b33da9; no voters:
I20260621 07:54:24.408625 26578 raft_consensus.cc:2781] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:54:24.408732 26578 raft_consensus.cc:482] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:54:24.408798 26578 raft_consensus.cc:3037] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:24.409850 26578 raft_consensus.cc:504] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:24.410231 26578 leader_election.cc:288] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [CANDIDATE]: Term 1 election: Requested vote from peers d01453ed24c64788a8db8af1b1e27947 (127.12.12.3:36083), d5092a43d05643658a7da11156b33da9 (127.12.12.2:38033)
I20260621 07:54:24.410641 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06c05a45c23436c8380fd8ab910d0c8" candidate_uuid: "435bf73c81a140d885d3bd500094f182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d01453ed24c64788a8db8af1b1e27947"
I20260621 07:54:24.410666 26375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06c05a45c23436c8380fd8ab910d0c8" candidate_uuid: "435bf73c81a140d885d3bd500094f182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5092a43d05643658a7da11156b33da9"
I20260621 07:54:24.410745 26509 raft_consensus.cc:3037] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:24.410812 26375 raft_consensus.cc:3037] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:24.411804 26509 raft_consensus.cc:2454] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 435bf73c81a140d885d3bd500094f182 in term 1.
I20260621 07:54:24.411895 26375 raft_consensus.cc:2454] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 435bf73c81a140d885d3bd500094f182 in term 1.
I20260621 07:54:24.412140 26176 leader_election.cc:302] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435bf73c81a140d885d3bd500094f182, d01453ed24c64788a8db8af1b1e27947; no voters:
I20260621 07:54:24.412317 26578 raft_consensus.cc:2781] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:54:24.412530 26578 raft_consensus.cc:686] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [term 1 LEADER]: Becoming Leader. State: Replica: 435bf73c81a140d885d3bd500094f182, State: Running, Role: LEADER
I20260621 07:54:24.412886 26578 consensus_queue.cc:237] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:24.416338 26094 catalog_manager.cc:5495] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 reported cstate change: term changed from 0 to 1, leader changed from <none> to 435bf73c81a140d885d3bd500094f182 (127.12.12.1). New cstate: current_term: 1 leader_uuid: "435bf73c81a140d885d3bd500094f182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } health_report { overall_health: HEALTHY } } }
W20260621 07:54:24.447373 26556 tablet.cc:2369] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:54:24.447499 26571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.443760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.12.12.1:45081)
0621 07:54:24.443915 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:24.443922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:24.443940 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:54:24.444023 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:24.444025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:24.444081 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:24.444196 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:54:24.444445 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.444975 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.445901 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.446169 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.446283 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.446627 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:54:24.447063 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:54:24.447228 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:24.447260 (+ 32us) server_negotiation.cc:299] Negotiation successful
0621 07:54:24.447316 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:54:24.453779 26509 raft_consensus.cc:1261] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947 [term 1 FOLLOWER]: Refusing update from remote peer 435bf73c81a140d885d3bd500094f182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:54:24.453780 26375 raft_consensus.cc:1261] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9 [term 1 FOLLOWER]: Refusing update from remote peer 435bf73c81a140d885d3bd500094f182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:54:24.454593 26583 consensus_queue.cc:1035] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:54:24.454809 26578 consensus_queue.cc:1035] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:54:24.461028 26586 mvcc.cc:205] Tried to move back new op lower bound from 7299188590395211776 to 7299188590238326784. Current Snapshot: MvccSnapshot[applied={T|T < 7299188590395211776}]
I20260621 07:54:24.463814 26588 mvcc.cc:205] Tried to move back new op lower bound from 7299188590395211776 to 7299188590238326784. Current Snapshot: MvccSnapshot[applied={T|T < 7299188590395211776}]
I20260621 07:54:24.464212 26587 mvcc.cc:205] Tried to move back new op lower bound from 7299188590395211776 to 7299188590238326784. Current Snapshot: MvccSnapshot[applied={T|T < 7299188590395211776}]
I20260621 07:54:24.470860 26095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:54:24.471042 26095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:54:24.473569 26095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09: Sending DeleteTablet for 3 replicas of tablet c06c05a45c23436c8380fd8ab910d0c8
I20260621 07:54:24.474427 26221 tablet_service.cc:1514] Processing DeleteTablet for tablet c06c05a45c23436c8380fd8ab910d0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:54:24 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:51386
I20260621 07:54:24.474434 26355 tablet_service.cc:1514] Processing DeleteTablet for tablet c06c05a45c23436c8380fd8ab910d0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:54:24 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:33498
I20260621 07:54:24.474525 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 26158
I20260621 07:54:24.474462 26489 tablet_service.cc:1514] Processing DeleteTablet for tablet c06c05a45c23436c8380fd8ab910d0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:54:24 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:46216
I20260621 07:54:24.476346 26597 tablet_replica.cc:330] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9: stopping tablet replica
I20260621 07:54:24.476686 26597 raft_consensus.cc:2229] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:54:24.477171 26597 raft_consensus.cc:2258] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:54:24.477540 26598 tablet_replica.cc:330] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947: stopping tablet replica
I20260621 07:54:24.477854 26598 raft_consensus.cc:2229] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:54:24.478138 26598 raft_consensus.cc:2258] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:54:24.479485 26597 ts_tablet_manager.cc:1905] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:54:24.481594 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 26292
W20260621 07:54:24.481585 26082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.1:45081 (error 108)
I20260621 07:54:24.482012 26598 ts_tablet_manager.cc:1905] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:54:24.484462 26598 ts_tablet_manager.cc:1918] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:54:24.484552 26598 log.cc:1196] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/c06c05a45c23436c8380fd8ab910d0c8
I20260621 07:54:24.484882 26598 ts_tablet_manager.cc:1939] T c06c05a45c23436c8380fd8ab910d0c8 P d01453ed24c64788a8db8af1b1e27947: Deleting consensus metadata
I20260621 07:54:24.485474 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.485144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:45081 (local address 127.0.0.1:51406)
0621 07:54:24.485302 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:24.485381 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260621 07:54:24.485683 26082 catalog_manager.cc:4584] TS 435bf73c81a140d885d3bd500094f182 (127.12.12.1:45081): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c06c05a45c23436c8380fd8ab910d0c8: Network error: Client connection negotiation failed: client connection to 127.12.12.1:45081: connect: Connection refused (error 111)
I20260621 07:54:24.486263 26082 catalog_manager.cc:4841] TS d01453ed24c64788a8db8af1b1e27947 (127.12.12.3:36083): tablet c06c05a45c23436c8380fd8ab910d0c8 (table test-table [id=b44d7ab11272422cab0d6a704d0668dd]) successfully deleted
I20260621 07:54:24.488278 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 26426
I20260621 07:54:24.489336 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.488368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:38033 (local address 127.0.0.1:33512)
0621 07:54:24.489170 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:24.489236 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.2:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":705}
I20260621 07:54:24.494779 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 26064
I20260621 07:54:24.501267 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:46527
--webserver_interface=127.12.12.62
--webserver_port=44553
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:46527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:54:24.616649 26600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:24.616969 26600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:24.617013 26600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:24.621249 26600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:54:24.621335 26600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:24.621377 26600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:24.621395 26600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:54:24.621410 26600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:54:24.627358 26600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:46527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:46527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=44553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:24.628517 26600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:24.629594 26600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:24.643399 26606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:24.643399 26607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:24.643399 26609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:24.643808 26600 server_base.cc:1034] running on GCE node
I20260621 07:54:24.644474 26600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:24.645771 26600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:24.646971 26600 hybrid_clock.cc:648] HybridClock initialized: now 1782028464646932 us; error 65 us; skew 500 ppm
Jun 21 07:54:24 dist-test-slave-lbpt krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028464, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:24.650220 26600 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:54:24.651314 26600 webserver.cc:460] Webserver started at http://127.12.12.62:44553/ using document root <none> and password file <none>
I20260621 07:54:24.651942 26600 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:24.652051 26600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:24.656272 26600 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260621 07:54:24.658890 26616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:24.660386 26600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 07:54:24.660558 26600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "b8b691e0131b432db6295d8af180bc09"
format_stamp: "Formatted at 2026-06-21 07:54:23 on dist-test-slave-lbpt"
server_key: "b1be7e0495772f8ae905a86eface7779"
server_key_iv: "9fd5ff728564a830bb3d38370862edaa"
server_key_version: "encryptionkey@0"
I20260621 07:54:24.661239 26600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:24.694058 26600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:24.698197 26600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:24.698508 26600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:24.706280 26600 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:46527
I20260621 07:54:24.706629 26668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:46527 every 8 connection(s)
I20260621 07:54:24.707736 26600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:54:24.709034 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 26600
I20260621 07:54:24.709514 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:45081
--local_ip_for_outbound_sockets=127.12.12.1
--tserver_master_addrs=127.12.12.62:46527
--webserver_port=36367
--webserver_interface=127.12.12.1
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:54:24.713969 26669 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:54:24.717020 26669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09: Bootstrap starting.
I20260621 07:54:24.740911 26669 log.cc:824] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:24.751981 26669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:54:24.752442 26669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09: Bootstrap complete.
I20260621 07:54:24.758195 26669 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } }
I20260621 07:54:24.758813 26669 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b691e0131b432db6295d8af180bc09, State: Initialized, Role: FOLLOWER
I20260621 07:54:24.759378 26669 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } }
I20260621 07:54:24.759526 26669 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:54:24.759593 26669 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:54:24.759685 26669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:54:24.762354 26669 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } }
I20260621 07:54:24.762779 26669 leader_election.cc:302] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8b691e0131b432db6295d8af180bc09; no voters:
I20260621 07:54:24.763139 26669 leader_election.cc:288] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 07:54:24.763298 26673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 07:54:24.763618 26673 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 2 LEADER]: Becoming Leader. State: Replica: b8b691e0131b432db6295d8af180bc09, State: Running, Role: LEADER
I20260621 07:54:24.763959 26673 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } }
I20260621 07:54:24.764442 26669 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:54:24.765519 26674 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b8b691e0131b432db6295d8af180bc09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } } }
I20260621 07:54:24.765519 26675 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b691e0131b432db6295d8af180bc09. Latest consensus state: current_term: 2 leader_uuid: "b8b691e0131b432db6295d8af180bc09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } } }
I20260621 07:54:24.765676 26674 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [sys.catalog]: This master's current role is: LEADER
I20260621 07:54:24.765676 26675 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [sys.catalog]: This master's current role is: LEADER
I20260621 07:54:24.766273 26682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:54:24.771183 26682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:54:24.771914 26682 catalog_manager.cc:1259] Loaded cluster ID: 8f61df329bd7409fa10b65bc5f2ae66b
I20260621 07:54:24.771981 26682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:54:24.773468 26682 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260621 07:54:24.778471 26682 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260621 07:54:24.784324 26682 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:54:24.785169 26682 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09: Loaded TSK: 0
I20260621 07:54:24.786448 26682 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:54:24.845855 26671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:24.846208 26671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:24.846307 26671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:24.850749 26671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:24.850826 26671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:24.850917 26671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:54:24.856729 26671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:45081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=36367
--tserver_master_addrs=127.12.12.62:46527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:24.857870 26671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:24.859021 26671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:24.870087 26696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:24.870210 26699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:24.870059 26697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:24.870645 26671 server_base.cc:1034] running on GCE node
I20260621 07:54:24.871130 26671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:24.871831 26671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:24.873044 26671 hybrid_clock.cc:648] HybridClock initialized: now 1782028464873036 us; error 38 us; skew 500 ppm
Jun 21 07:54:24 dist-test-slave-lbpt krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028464, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:24.880174 26671 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:54:24.881305 26671 webserver.cc:460] Webserver started at http://127.12.12.1:36367/ using document root <none> and password file <none>
I20260621 07:54:24.881919 26671 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:24.881996 26671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:24.886224 26671 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:54:24.888823 26706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:24.890226 26671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:54:24.890374 26671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "435bf73c81a140d885d3bd500094f182"
format_stamp: "Formatted at 2026-06-21 07:54:23 on dist-test-slave-lbpt"
server_key: "952c344175f7f5645a1fcf26bcf58682"
server_key_iv: "f2a7e03ae8be226d7ec9b7f09e2a3bc6"
server_key_version: "encryptionkey@0"
I20260621 07:54:24.891069 26671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:24.905926 26671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:24.909411 26671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:24.909624 26671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:24.910257 26671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:24.911527 26713 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 07:54:24.915804 26671 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 07:54:24.915877 26671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260621 07:54:24.915930 26671 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 07:54:24.917781 26671 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 07:54:24.917843 26671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260621 07:54:24.918115 26713 tablet_bootstrap.cc:492] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182: Bootstrap starting.
I20260621 07:54:24.929092 26671 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:45081
I20260621 07:54:24.929144 26820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:45081 every 8 connection(s)
I20260621 07:54:24.930444 26671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:54:24.938118 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 26671
I20260621 07:54:24.938689 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:38033
--local_ip_for_outbound_sockets=127.12.12.2
--tserver_master_addrs=127.12.12.62:46527
--webserver_port=33763
--webserver_interface=127.12.12.2
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:54:24.939039 26713 log.cc:824] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:54:24 dist-test-slave-lbpt krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028464, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:24.947996 26713 tablet_bootstrap.cc:492] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:54:24.948493 26713 tablet_bootstrap.cc:492] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182: Bootstrap complete.
I20260621 07:54:24.949892 26713 ts_tablet_manager.cc:1397] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.016s
I20260621 07:54:24.950682 26823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.932707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:50745 (local address 127.12.12.62:46527)
0621 07:54:24.933085 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:24.933106 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:24.933472 (+ 366us) server_negotiation.cc:407] Connection header received
0621 07:54:24.934702 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:24.934714 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:24.935032 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:24.935382 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:24.936540 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.937540 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.939574 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.940330 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.940536 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.943949 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:24.943960 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:24.943982 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:24.944028 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:24.946543 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:24.947209 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:24.947215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:24.947221 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:24.947381 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:24.947812 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:24.947816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:24.947817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:24.948333 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:24.948485 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:24.949238 (+ 753us) server_negotiation.cc:299] Negotiation successful
0621 07:54:24.949574 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":129,"threads_started":1}
I20260621 07:54:24.951468 26824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.932736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:50745)
0621 07:54:24.933297 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:24.933341 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:24.934222 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:24.935434 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:24.935442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:24.935884 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:24.936353 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:24.936367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.937701 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.937712 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:24.939361 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:24.939378 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.940598 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.940601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:24.940814 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.941616 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:24.941640 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:24.943694 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:24.946679 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:24.946684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:24.946698 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:24.947062 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:24.947524 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:24.947527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:24.947529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:24.947693 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:24.948476 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:24.948485 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:24.948852 (+ 367us) client_negotiation.cc:769] Sending connection context
0621 07:54:24.949096 (+ 244us) client_negotiation.cc:240] Negotiation successful
0621 07:54:24.949509 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":171,"threads_started":1}
I20260621 07:54:24.952922 26713 raft_consensus.cc:348] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:24.953357 26713 raft_consensus.cc:729] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435bf73c81a140d885d3bd500094f182, State: Initialized, Role: FOLLOWER
I20260621 07:54:24.953961 26821 heartbeater.cc:347] Connected to a master server at 127.12.12.62:46527
I20260621 07:54:24.953927 26713 consensus_queue.cc:260] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:24.954200 26821 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:24.954788 26821 heartbeater.cc:511] Master 127.12.12.62:46527 requested a full tablet report, sending...
I20260621 07:54:24.955049 26713 ts_tablet_manager.cc:1428] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:54:24.957419 26628 ts_manager.cc:194] Registered new tserver with Master: 435bf73c81a140d885d3bd500094f182 (127.12.12.1:45081)
I20260621 07:54:24.961174 26628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:50745
I20260621 07:54:24.964069 26821 heartbeater.cc:503] Master 127.12.12.62:46527 was elected leader, sending a full tablet report...
Jun 21 07:54:24 dist-test-slave-lbpt krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028464, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for kudu/127.12.12.1@KRBTEST.COM
I20260621 07:54:24.972047 26828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.960484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:45081 (local address 127.0.0.1:51420)
0621 07:54:24.961245 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:24.961292 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:24.961487 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:24.962045 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:24.962051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:24.962547 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:24.962679 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:24.962687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.963995 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.963999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:24.964885 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:24.964915 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.965547 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.965551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:24.965668 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.966450 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:24.966470 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:24.968092 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:24.970153 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:24.970161 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:24.970166 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:24.970396 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:24.970842 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:24.970845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:24.970846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:24.970933 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:24.971508 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:24.971513 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:24.971629 (+ 116us) client_negotiation.cc:769] Sending connection context
0621 07:54:24.971744 (+ 115us) client_negotiation.cc:240] Negotiation successful
0621 07:54:24.971909 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":136,"threads_started":1}
I20260621 07:54:24.972211 26829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:24.960585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.12.12.1:45081)
0621 07:54:24.960976 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:24.960986 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:24.961381 (+ 395us) server_negotiation.cc:407] Connection header received
0621 07:54:24.961676 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:24.961682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:24.961893 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:24.962041 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:24.962824 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.963851 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.965015 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:24.965410 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:24.965591 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:24.968244 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:24.968253 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:24.968266 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:24.968293 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:24.970024 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:24.970615 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:24.970622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:24.970623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:24.970720 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:24.971044 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:24.971047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:24.971049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:24.971379 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:24.971511 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:24.971867 (+ 356us) server_negotiation.cc:299] Negotiation successful
0621 07:54:24.972042 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":117,"threads_started":1}
I20260621 07:54:24.973173 26755 tablet_service.cc:1514] Processing DeleteTablet for tablet c06c05a45c23436c8380fd8ab910d0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:54:24 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:51420
I20260621 07:54:24.974167 26830 tablet_replica.cc:330] stopping tablet replica
I20260621 07:54:24.974521 26830 raft_consensus.cc:2229] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:54:24.974781 26830 raft_consensus.cc:2258] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:54:24.975751 26830 ts_tablet_manager.cc:1905] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:54:24.979717 26830 ts_tablet_manager.cc:1918] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:54:24.979837 26830 log.cc:1196] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal/wals/c06c05a45c23436c8380fd8ab910d0c8
I20260621 07:54:24.980110 26830 ts_tablet_manager.cc:1939] T c06c05a45c23436c8380fd8ab910d0c8 P 435bf73c81a140d885d3bd500094f182: Deleting consensus metadata
I20260621 07:54:24.981817 26619 catalog_manager.cc:4841] TS 435bf73c81a140d885d3bd500094f182 (127.12.12.1:45081): tablet c06c05a45c23436c8380fd8ab910d0c8 (table test-table [id=b44d7ab11272422cab0d6a704d0668dd]) successfully deleted
W20260621 07:54:25.061290 26826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:25.061620 26826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:25.061703 26826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:25.066176 26826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:25.066267 26826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:25.066367 26826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:54:25.071748 26826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:38033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=33763
--tserver_master_addrs=127.12.12.62:46527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:25.072868 26826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:25.073943 26826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:25.083920 26838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:25.083914 26835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:25.083914 26836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:25.084035 26826 server_base.cc:1034] running on GCE node
I20260621 07:54:25.084591 26826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:25.085233 26826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:25.086400 26826 hybrid_clock.cc:648] HybridClock initialized: now 1782028465086358 us; error 55 us; skew 500 ppm
Jun 21 07:54:25 dist-test-slave-lbpt krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:25.089183 26826 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:54:25.090106 26826 webserver.cc:460] Webserver started at http://127.12.12.2:33763/ using document root <none> and password file <none>
I20260621 07:54:25.090730 26826 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:25.090811 26826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:25.094959 26826 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:54:25.097530 26845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:25.099018 26826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:54:25.099140 26826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "d5092a43d05643658a7da11156b33da9"
format_stamp: "Formatted at 2026-06-21 07:54:23 on dist-test-slave-lbpt"
server_key: "82a4c47d4c9c0a25f7afa5889f762a99"
server_key_iv: "ed4e0725d31c950ea170a13c117bf923"
server_key_version: "encryptionkey@0"
I20260621 07:54:25.099700 26826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:25.120790 26826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:25.123831 26826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:25.124033 26826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:25.124660 26826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:25.125774 26852 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260621 07:54:25.129601 26852 ts_tablet_manager.cc:1852] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260621 07:54:25.129995 26852 ts_tablet_manager.cc:1905] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:54:25.133775 26852 ts_tablet_manager.cc:1918] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:54:25.133869 26852 log.cc:1196] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/c06c05a45c23436c8380fd8ab910d0c8
I20260621 07:54:25.134191 26852 ts_tablet_manager.cc:1939] T c06c05a45c23436c8380fd8ab910d0c8 P d5092a43d05643658a7da11156b33da9: Deleting consensus metadata
I20260621 07:54:25.134464 26826 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260621 07:54:25.134531 26826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260621 07:54:25.134599 26826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:54:25.134632 26826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:25.143999 26826 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:38033
I20260621 07:54:25.144025 26959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:38033 every 8 connection(s)
I20260621 07:54:25.144909 26826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:54:25.146682 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 26826
I20260621 07:54:25.147063 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:36083
--local_ip_for_outbound_sockets=127.12.12.3
--tserver_master_addrs=127.12.12.62:46527
--webserver_port=38413
--webserver_interface=127.12.12.3
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:54:25 dist-test-slave-lbpt krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:25.159968 26823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.146791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:58471 (local address 127.12.12.62:46527)
0621 07:54:25.146937 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:25.146941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:25.148059 (+ 1118us) server_negotiation.cc:407] Connection header received
0621 07:54:25.149218 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:25.149224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:25.149291 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:25.149409 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:25.150607 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.151523 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.152638 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.153083 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.153208 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:25.155794 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:25.155801 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:25.155807 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:25.155846 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:25.157736 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:25.158306 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:25.158309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:25.158311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:25.158401 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:25.158828 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:25.158832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:25.158834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:25.159066 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:25.159167 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:25.159659 (+ 492us) server_negotiation.cc:299] Negotiation successful
0621 07:54:25.159795 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:54:25.160787 26962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.147325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:58471)
0621 07:54:25.147879 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:25.147937 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:25.148975 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:25.149557 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:25.149564 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:25.149933 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:25.150392 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:25.150404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.151655 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.151658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:25.152503 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:25.152511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.153248 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.153250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:25.153398 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:25.154101 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:25.154116 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:25.155609 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:25.157862 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:25.157867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:25.157880 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:25.158197 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:25.158517 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:25.158521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:25.158523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:25.158727 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:25.159170 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:25.159176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:25.159422 (+ 246us) client_negotiation.cc:769] Sending connection context
0621 07:54:25.159584 (+ 162us) client_negotiation.cc:240] Negotiation successful
0621 07:54:25.159853 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":137,"threads_started":1}
I20260621 07:54:25.162020 26960 heartbeater.cc:347] Connected to a master server at 127.12.12.62:46527
I20260621 07:54:25.162273 26960 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:25.162834 26960 heartbeater.cc:511] Master 127.12.12.62:46527 requested a full tablet report, sending...
I20260621 07:54:25.163980 26628 ts_manager.cc:194] Registered new tserver with Master: d5092a43d05643658a7da11156b33da9 (127.12.12.2:38033)
I20260621 07:54:25.164762 26628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:58471
W20260621 07:54:25.263734 26963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:25.264007 26963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:25.264109 26963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:25.268695 26963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:25.268755 26963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:25.268847 26963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:54:25.274228 26963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:36083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=38413
--tserver_master_addrs=127.12.12.62:46527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:25.275373 26963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:25.276443 26963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:54:25.286767 26963 server_base.cc:1034] running on GCE node
W20260621 07:54:25.286813 26971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:25.286711 26968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:25.286705 26969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:25.287369 26963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:25.288035 26963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:25.289224 26963 hybrid_clock.cc:648] HybridClock initialized: now 1782028465289206 us; error 33 us; skew 500 ppm
Jun 21 07:54:25 dist-test-slave-lbpt krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:25.292165 26963 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:54:25.293082 26963 webserver.cc:460] Webserver started at http://127.12.12.3:38413/ using document root <none> and password file <none>
I20260621 07:54:25.293663 26963 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:25.293784 26963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:25.297819 26963 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:54:25.300043 26978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:25.301434 26963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:54:25.301631 26963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "d01453ed24c64788a8db8af1b1e27947"
format_stamp: "Formatted at 2026-06-21 07:54:24 on dist-test-slave-lbpt"
server_key: "dac034e32b1454a52680901fe4ac823d"
server_key_iv: "e1af86d3c2ef7e2b7f88b009892a81c0"
server_key_version: "encryptionkey@0"
I20260621 07:54:25.302202 26963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:25.328451 26963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:25.331704 26963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:25.331912 26963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:25.332536 26963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:25.333357 26963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:54:25.333429 26963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:25.333503 26963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:54:25.333556 26963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:25.342402 26963 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:36083
I20260621 07:54:25.342501 27091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:36083 every 8 connection(s)
I20260621 07:54:25.343433 26963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:54:25.344316 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 26963
I20260621 07:54:25.353811 26823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.345046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.12.12.62:46527)
0621 07:54:25.345211 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:25.345216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:25.345413 (+ 197us) server_negotiation.cc:407] Connection header received
0621 07:54:25.345534 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:25.345540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:25.345604 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:25.345688 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:25.346773 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.347569 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.348558 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.348886 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.348991 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:25.350295 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:25.350299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:25.350301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:25.350325 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:25.352068 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:25.352536 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:25.352539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:25.352540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:25.352614 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:25.352898 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:25.352901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:25.352902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:25.353115 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:25.353328 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:25.353446 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 07:54:25.353606 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 21 07:54:25 dist-test-slave-lbpt krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:25.359067 27095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.345746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:57127 (local address 127.12.12.62:46527)
0621 07:54:25.345960 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:25.345963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:25.347070 (+ 1107us) server_negotiation.cc:407] Connection header received
0621 07:54:25.348614 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:25.348617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:25.348663 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:25.348728 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:25.350081 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.350775 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.352465 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.352786 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.352896 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:25.355571 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:25.355575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:25.355577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:25.355600 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:25.356983 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:25.357514 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:25.357518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:25.357520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:25.357599 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:25.357942 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:25.357945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:25.357947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:25.358123 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:25.358231 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:25.358781 (+ 550us) server_negotiation.cc:299] Negotiation successful
0621 07:54:25.358897 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
I20260621 07:54:25.359903 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.346308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:57127)
0621 07:54:25.346878 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:25.346933 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:25.348410 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:25.348904 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:25.348915 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:25.349398 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:25.349927 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:25.349941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.350917 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.350923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:25.352342 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:25.352351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.352951 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.352957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:25.353184 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:25.353781 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:25.353800 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:25.355409 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:25.357113 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:25.357119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:25.357133 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:25.357395 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:25.357690 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:25.357695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:25.357697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:25.357838 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:25.358230 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:25.358236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:25.358545 (+ 309us) client_negotiation.cc:769] Sending connection context
0621 07:54:25.358743 (+ 198us) client_negotiation.cc:240] Negotiation successful
0621 07:54:25.359075 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":129,"threads_started":1}
I20260621 07:54:25.361083 27092 heartbeater.cc:347] Connected to a master server at 127.12.12.62:46527
I20260621 07:54:25.361338 27092 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:25.361840 27092 heartbeater.cc:511] Master 127.12.12.62:46527 requested a full tablet report, sending...
I20260621 07:54:25.362958 26628 ts_manager.cc:194] Registered new tserver with Master: d01453ed24c64788a8db8af1b1e27947 (127.12.12.3:36083)
I20260621 07:54:25.363545 26628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:57127
I20260621 07:54:25.367626 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:54:25.371960 27095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.368254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.12.12.62:46527)
0621 07:54:25.368400 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:25.368405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:25.368585 (+ 180us) server_negotiation.cc:407] Connection header received
0621 07:54:25.368716 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:25.368718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:25.368813 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:25.368895 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:54:25.369121 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.369649 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.370579 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.370806 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.370888 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:25.371204 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:54:25.371560 (+ 356us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:54:25.371672 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:25.371735 (+ 63us) server_negotiation.cc:299] Negotiation successful
0621 07:54:25.371785 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 07:54:25.372536 26628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:54:25.375813 26628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:54:25.383921 26755 tablet_service.cc:1467] Processing CreateTablet for tablet 8208d6302b194e57a4a6f661bc90e56a (DEFAULT_TABLE table=test-table [id=cd249aa109734b58866b67a63613ecec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:54:25.384626 26755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8208d6302b194e57a4a6f661bc90e56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:25.387708 26713 tablet_bootstrap.cc:492] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182: Bootstrap starting.
I20260621 07:54:25.388793 26713 tablet_bootstrap.cc:654] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:25.388933 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.383264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.12.12.3:36083)
0621 07:54:25.384005 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:25.384020 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:25.384038 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:54:25.384100 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:25.384104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:25.384251 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:25.384417 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:25.384871 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.386039 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.387292 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.388257 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.388367 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:25.388523 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:25.388657 (+ 134us) server_negotiation.cc:299] Negotiation successful
0621 07:54:25.388760 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":177,"threads_started":1}
I20260621 07:54:25.389214 27101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.383032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:36083 (local address 127.0.0.1:46224)
0621 07:54:25.383305 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:25.383328 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:25.383421 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:25.384618 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:25.384622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:25.384638 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:25.384736 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:25.384746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.386217 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.386220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:25.387150 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:25.387160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.388371 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.388375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:25.388465 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:25.388520 (+ 55us) client_negotiation.cc:769] Sending connection context
0621 07:54:25.388623 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 07:54:25.388731 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":113,"threads_started":1}
I20260621 07:54:25.389428 26828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.382648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:38033 (local address 127.0.0.1:33526)
0621 07:54:25.382780 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:25.382797 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:25.382876 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:25.384539 (+ 1663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:25.384542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:25.384560 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:25.384676 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:25.384684 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.386192 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.386198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:25.387470 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:25.387484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.389001 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.389005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:25.389108 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:25.389131 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 07:54:25.389185 (+ 54us) client_negotiation.cc:240] Negotiation successful
0621 07:54:25.389265 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":30}
I20260621 07:54:25.390223 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.383542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33526 (local address 127.12.12.2:38033)
0621 07:54:25.383868 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:25.383881 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:25.383906 (+ 25us) server_negotiation.cc:407] Connection header received
0621 07:54:25.383985 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:25.383991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:25.384402 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:25.384547 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:25.384783 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.386026 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.388013 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.388852 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.389782 (+ 930us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:25.389923 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:25.389984 (+ 61us) server_negotiation.cc:299] Negotiation successful
0621 07:54:25.390071 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":130,"threads_started":1}
I20260621 07:54:25.390367 26713 tablet_bootstrap.cc:492] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182: No bootstrap required, opened a new log
I20260621 07:54:25.390489 26713 ts_tablet_manager.cc:1397] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260621 07:54:25.390959 26713 raft_consensus.cc:348] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:25.391117 26713 raft_consensus.cc:374] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:25.391160 26713 raft_consensus.cc:729] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435bf73c81a140d885d3bd500094f182, State: Initialized, Role: FOLLOWER
I20260621 07:54:25.391323 26713 consensus_queue.cc:260] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:25.391706 26894 tablet_service.cc:1467] Processing CreateTablet for tablet 8208d6302b194e57a4a6f661bc90e56a (DEFAULT_TABLE table=test-table [id=cd249aa109734b58866b67a63613ecec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:54:25.392182 26713 ts_tablet_manager.cc:1428] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:54:25.392738 26894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8208d6302b194e57a4a6f661bc90e56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:25.393086 27026 tablet_service.cc:1467] Processing CreateTablet for tablet 8208d6302b194e57a4a6f661bc90e56a (DEFAULT_TABLE table=test-table [id=cd249aa109734b58866b67a63613ecec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:54:25.394150 27026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8208d6302b194e57a4a6f661bc90e56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:25.397580 26852 tablet_bootstrap.cc:492] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9: Bootstrap starting.
I20260621 07:54:25.399784 26852 tablet_bootstrap.cc:654] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:25.400501 26852 log.cc:824] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:25.401268 27106 tablet_bootstrap.cc:492] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: Bootstrap starting.
I20260621 07:54:25.402266 26852 tablet_bootstrap.cc:492] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9: No bootstrap required, opened a new log
I20260621 07:54:25.402485 26852 ts_tablet_manager.cc:1397] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:54:25.403357 27106 tablet_bootstrap.cc:654] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:25.404659 27106 log.cc:824] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:25.406073 26852 raft_consensus.cc:348] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:25.406371 26852 raft_consensus.cc:374] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:25.406461 26852 raft_consensus.cc:729] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5092a43d05643658a7da11156b33da9, State: Initialized, Role: FOLLOWER
I20260621 07:54:25.406736 27106 tablet_bootstrap.cc:492] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: No bootstrap required, opened a new log
I20260621 07:54:25.406929 27106 ts_tablet_manager.cc:1397] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:54:25.406942 26852 consensus_queue.cc:260] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:25.407655 26960 heartbeater.cc:503] Master 127.12.12.62:46527 was elected leader, sending a full tablet report...
I20260621 07:54:25.407941 26852 ts_tablet_manager.cc:1428] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260621 07:54:25.410705 27106 raft_consensus.cc:348] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:25.410933 27106 raft_consensus.cc:374] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:25.410993 27106 raft_consensus.cc:729] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d01453ed24c64788a8db8af1b1e27947, State: Initialized, Role: FOLLOWER
I20260621 07:54:25.411427 27106 consensus_queue.cc:260] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:25.412078 27092 heartbeater.cc:503] Master 127.12.12.62:46527 was elected leader, sending a full tablet report...
I20260621 07:54:25.412372 27106 ts_tablet_manager.cc:1428] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260621 07:54:25.431752 26822 tablet.cc:2369] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:54:25.465828 27109 raft_consensus.cc:482] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:54:25.466025 27109 raft_consensus.cc:504] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:25.467370 27109 leader_election.cc:288] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5092a43d05643658a7da11156b33da9 (127.12.12.2:38033), 435bf73c81a140d885d3bd500094f182 (127.12.12.1:45081)
I20260621 07:54:25.471385 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.467566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:38033 (local address 127.12.12.3:60619)
0621 07:54:25.467689 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:25.467707 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:25.467801 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:25.468326 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:25.468329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:25.468345 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:25.468481 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:25.468487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.469302 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.469305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:25.470157 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:25.470166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.470937 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.470941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:25.471060 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:25.471085 (+ 25us) client_negotiation.cc:769] Sending connection context
0621 07:54:25.471161 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 07:54:25.471236 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260621 07:54:25.471432 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.467984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:60619 (local address 127.12.12.2:38033)
0621 07:54:25.468119 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:25.468123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:25.468135 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:54:25.468182 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:25.468184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:25.468227 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:25.468292 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:25.468652 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.469186 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.470302 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.470812 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.470901 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:25.470952 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:25.471208 (+ 256us) server_negotiation.cc:299] Negotiation successful
0621 07:54:25.471292 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 07:54:25.471557 27110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.467638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:45081 (local address 127.12.12.3:41217)
0621 07:54:25.468027 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:25.468037 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:25.468148 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:25.468457 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:25.468460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:25.468476 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:25.468638 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:25.468645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.469417 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.469420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:25.470270 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:25.470283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.471101 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.471103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:25.471243 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:25.471265 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:54:25.471357 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 07:54:25.471422 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":104,"threads_started":1}
I20260621 07:54:25.471606 26829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.467793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:41217 (local address 127.12.12.1:45081)
0621 07:54:25.467946 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:25.467951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:25.468124 (+ 173us) server_negotiation.cc:407] Connection header received
0621 07:54:25.468235 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:25.468238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:25.468328 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:25.468430 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:25.468755 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.469295 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.470403 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.470962 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.471071 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:25.471246 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:25.471370 (+ 124us) server_negotiation.cc:299] Negotiation successful
0621 07:54:25.471453 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 07:54:25.472164 26775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8208d6302b194e57a4a6f661bc90e56a" candidate_uuid: "d01453ed24c64788a8db8af1b1e27947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435bf73c81a140d885d3bd500094f182" is_pre_election: true
I20260621 07:54:25.472296 26914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8208d6302b194e57a4a6f661bc90e56a" candidate_uuid: "d01453ed24c64788a8db8af1b1e27947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5092a43d05643658a7da11156b33da9" is_pre_election: true
I20260621 07:54:25.472534 26775 raft_consensus.cc:2454] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d01453ed24c64788a8db8af1b1e27947 in term 0.
I20260621 07:54:25.472622 26914 raft_consensus.cc:2454] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d01453ed24c64788a8db8af1b1e27947 in term 0.
I20260621 07:54:25.473002 26981 leader_election.cc:302] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435bf73c81a140d885d3bd500094f182, d01453ed24c64788a8db8af1b1e27947; no voters:
I20260621 07:54:25.473294 27109 raft_consensus.cc:2781] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:54:25.473372 27109 raft_consensus.cc:482] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:54:25.473403 27109 raft_consensus.cc:3037] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:25.474398 27109 raft_consensus.cc:504] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:25.474821 27109 leader_election.cc:288] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [CANDIDATE]: Term 1 election: Requested vote from peers d5092a43d05643658a7da11156b33da9 (127.12.12.2:38033), 435bf73c81a140d885d3bd500094f182 (127.12.12.1:45081)
I20260621 07:54:25.475164 26914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8208d6302b194e57a4a6f661bc90e56a" candidate_uuid: "d01453ed24c64788a8db8af1b1e27947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5092a43d05643658a7da11156b33da9"
I20260621 07:54:25.475217 26775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8208d6302b194e57a4a6f661bc90e56a" candidate_uuid: "d01453ed24c64788a8db8af1b1e27947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435bf73c81a140d885d3bd500094f182"
I20260621 07:54:25.475278 26914 raft_consensus.cc:3037] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:25.475365 26775 raft_consensus.cc:3037] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:25.476343 26914 raft_consensus.cc:2454] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d01453ed24c64788a8db8af1b1e27947 in term 1.
I20260621 07:54:25.476424 26775 raft_consensus.cc:2454] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d01453ed24c64788a8db8af1b1e27947 in term 1.
I20260621 07:54:25.476639 26980 leader_election.cc:302] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d01453ed24c64788a8db8af1b1e27947, d5092a43d05643658a7da11156b33da9; no voters:
I20260621 07:54:25.476866 27109 raft_consensus.cc:2781] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:54:25.477088 27109 raft_consensus.cc:686] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [term 1 LEADER]: Becoming Leader. State: Replica: d01453ed24c64788a8db8af1b1e27947, State: Running, Role: LEADER
I20260621 07:54:25.477447 27109 consensus_queue.cc:237] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:25.480769 26631 catalog_manager.cc:5495] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 reported cstate change: term changed from 0 to 1, leader changed from <none> to d01453ed24c64788a8db8af1b1e27947 (127.12.12.3). New cstate: current_term: 1 leader_uuid: "d01453ed24c64788a8db8af1b1e27947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } health_report { overall_health: UNKNOWN } } }
I20260621 07:54:25.492228 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.488630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.12.12.3:36083)
0621 07:54:25.488765 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:25.488770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:25.488784 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:54:25.488964 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:25.488967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:25.489026 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:25.489135 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:54:25.489365 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.489879 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.490806 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:25.491037 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:25.491146 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:25.491434 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:54:25.491833 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:54:25.491968 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:25.492007 (+ 39us) server_negotiation.cc:299] Negotiation successful
0621 07:54:25.492061 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:54:25.498030 26775 raft_consensus.cc:1261] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 1 FOLLOWER]: Refusing update from remote peer d01453ed24c64788a8db8af1b1e27947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:54:25.498031 26914 raft_consensus.cc:1261] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [term 1 FOLLOWER]: Refusing update from remote peer d01453ed24c64788a8db8af1b1e27947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:54:25.498814 27111 consensus_queue.cc:1035] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:54:25.499095 27109 consensus_queue.cc:1035] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:54:25.505791 27114 mvcc.cc:205] Tried to move back new op lower bound from 7299188594673205248 to 7299188594598449152. Current Snapshot: MvccSnapshot[applied={T|T < 7299188594673205248}]
I20260621 07:54:25.506489 27115 mvcc.cc:205] Tried to move back new op lower bound from 7299188594673205248 to 7299188594598449152. Current Snapshot: MvccSnapshot[applied={T|T < 7299188594673205248}]
I20260621 07:54:25.508767 27116 mvcc.cc:205] Tried to move back new op lower bound from 7299188594673205248 to 7299188594598449152. Current Snapshot: MvccSnapshot[applied={T|T < 7299188594673205248}]
I20260621 07:54:25.514052 26631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:54:25.514204 26631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:54:25.516669 26631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09: Sending DeleteTablet for 3 replicas of tablet 8208d6302b194e57a4a6f661bc90e56a
I20260621 07:54:25.517354 26894 tablet_service.cc:1514] Processing DeleteTablet for tablet 8208d6302b194e57a4a6f661bc90e56a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:54:25 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:33526
I20260621 07:54:25.517387 26755 tablet_service.cc:1514] Processing DeleteTablet for tablet 8208d6302b194e57a4a6f661bc90e56a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:54:25 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:51420
I20260621 07:54:25.517395 27026 tablet_service.cc:1514] Processing DeleteTablet for tablet 8208d6302b194e57a4a6f661bc90e56a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:54:25 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:46224
I20260621 07:54:25.517863 27124 tablet_replica.cc:330] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9: stopping tablet replica
I20260621 07:54:25.517892 27125 tablet_replica.cc:330] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: stopping tablet replica
I20260621 07:54:25.518167 27125 raft_consensus.cc:2229] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:54:25.518167 27124 raft_consensus.cc:2229] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:54:25.518270 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 26671
I20260621 07:54:25.518488 27124 raft_consensus.cc:2258] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:54:25.519032 27125 raft_consensus.cc:2258] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:54:25.520488 27125 ts_tablet_manager.cc:1905] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:54:25.523367 27125 ts_tablet_manager.cc:1918] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:54:25.523483 27125 log.cc:1196] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/8208d6302b194e57a4a6f661bc90e56a
I20260621 07:54:25.524281 27125 ts_tablet_manager.cc:1939] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: Deleting consensus metadata
I20260621 07:54:25.524969 27124 ts_tablet_manager.cc:1905] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260621 07:54:25.524976 26619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.1:45081 (error 108)
I20260621 07:54:25.525611 26619 catalog_manager.cc:4841] TS d01453ed24c64788a8db8af1b1e27947 (127.12.12.3:36083): tablet 8208d6302b194e57a4a6f661bc90e56a (table test-table [id=cd249aa109734b58866b67a63613ecec]) successfully deleted
I20260621 07:54:25.526141 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 26826
I20260621 07:54:25.528623 26828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.528278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:45081 (local address 127.0.0.1:51436)
0621 07:54:25.528461 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:25.528534 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260621 07:54:25.528810 26619 catalog_manager.cc:4584] TS 435bf73c81a140d885d3bd500094f182 (127.12.12.1:45081): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8208d6302b194e57a4a6f661bc90e56a: Network error: Client connection negotiation failed: client connection to 127.12.12.1:45081: connect: Connection refused (error 111)
I20260621 07:54:25.532161 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 26963
I20260621 07:54:25.533948 26828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:25.532625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:38033 (local address 127.0.0.1:33540)
0621 07:54:25.533792 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:25.533879 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.2:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1059}
I20260621 07:54:25.537367 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 26600
I20260621 07:54:25.544096 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:46527
--webserver_interface=127.12.12.62
--webserver_port=44553
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:46527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:54:25.655349 27127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:25.655670 27127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:25.655752 27127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:25.659830 27127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:54:25.659919 27127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:25.659956 27127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:25.660017 27127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:54:25.660058 27127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:54:25.665355 27127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:46527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:46527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=44553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:25.666612 27127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:25.667693 27127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:25.676981 27132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:25.677014 27127 server_base.cc:1034] running on GCE node
W20260621 07:54:25.676970 27133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:25.676976 27135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:25.677702 27127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:25.678872 27127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:25.680029 27127 hybrid_clock.cc:648] HybridClock initialized: now 1782028465680008 us; error 35 us; skew 500 ppm
Jun 21 07:54:25 dist-test-slave-lbpt krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:25.682749 27127 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:54:25.683562 27127 webserver.cc:460] Webserver started at http://127.12.12.62:44553/ using document root <none> and password file <none>
I20260621 07:54:25.684080 27127 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:25.684140 27127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:25.687942 27127 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:54:25.689970 27142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:25.691314 27127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:54:25.691439 27127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "b8b691e0131b432db6295d8af180bc09"
format_stamp: "Formatted at 2026-06-21 07:54:23 on dist-test-slave-lbpt"
server_key: "b1be7e0495772f8ae905a86eface7779"
server_key_iv: "9fd5ff728564a830bb3d38370862edaa"
server_key_version: "encryptionkey@0"
I20260621 07:54:25.691982 27127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:25.708339 27127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:25.711699 27127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:25.711926 27127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:25.718977 27127 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:46527
I20260621 07:54:25.719264 27194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:46527 every 8 connection(s)
I20260621 07:54:25.720136 27127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:54:25.724620 27195 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:54:25.726526 27195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09: Bootstrap starting.
I20260621 07:54:25.730521 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 27127
I20260621 07:54:25.731004 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:45081
--local_ip_for_outbound_sockets=127.12.12.1
--tserver_master_addrs=127.12.12.62:46527
--webserver_port=36367
--webserver_interface=127.12.12.1
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:54:25.745199 27195 log.cc:824] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:25.756743 27195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:54:25.757151 27195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09: Bootstrap complete.
I20260621 07:54:25.761242 27195 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } }
I20260621 07:54:25.761662 27195 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b691e0131b432db6295d8af180bc09, State: Initialized, Role: FOLLOWER
I20260621 07:54:25.762153 27195 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } }
I20260621 07:54:25.762311 27195 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:54:25.762398 27195 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:54:25.762538 27195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 2 FOLLOWER]: Advancing to term 3
I20260621 07:54:25.764408 27195 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } }
I20260621 07:54:25.764742 27195 leader_election.cc:302] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8b691e0131b432db6295d8af180bc09; no voters:
I20260621 07:54:25.765082 27195 leader_election.cc:288] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 07:54:25.765165 27199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 3 FOLLOWER]: Leader election won for term 3
I20260621 07:54:25.765380 27199 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [term 3 LEADER]: Becoming Leader. State: Replica: b8b691e0131b432db6295d8af180bc09, State: Running, Role: LEADER
I20260621 07:54:25.765671 27199 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } }
I20260621 07:54:25.766322 27195 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:54:25.766654 27199 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b691e0131b432db6295d8af180bc09. Latest consensus state: current_term: 3 leader_uuid: "b8b691e0131b432db6295d8af180bc09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } } }
I20260621 07:54:25.766784 27199 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [sys.catalog]: This master's current role is: LEADER
I20260621 07:54:25.767138 27203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:54:25.767158 27200 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b8b691e0131b432db6295d8af180bc09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b691e0131b432db6295d8af180bc09" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 46527 } } }
I20260621 07:54:25.767388 27200 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b691e0131b432db6295d8af180bc09 [sys.catalog]: This master's current role is: LEADER
I20260621 07:54:25.772382 27203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:54:25.773211 27203 catalog_manager.cc:1259] Loaded cluster ID: 8f61df329bd7409fa10b65bc5f2ae66b
I20260621 07:54:25.773293 27203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260621 07:54:25.774088 27203 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fecc38219ad google::LogMessage::Fail() at ??:0
@ 0x7fecc382488a google::LogMessage::SendToLog() at ??:0
@ 0x7fecc38214de google::LogMessage::Flush() at ??:0
@ 0x7fecc3822a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fecce164fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fecce165d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fecce162579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fecce1a6640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fecc5a733d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fecc5a73c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fecc5a753f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fecc5a64512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fecc70a86db start_thread at ??:0
@ 0x7fecc231f61f clone at ??:0
W20260621 07:54:25.861050 27197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:25.861357 27197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:25.861449 27197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:25.865625 27197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:25.865787 27197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:25.865907 27197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:54:25.871457 27197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:45081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=36367
--tserver_master_addrs=127.12.12.62:46527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:25.872570 27197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:25.873607 27197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:25.883366 27236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:25.883388 27237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:25.883404 27239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:25.883728 27197 server_base.cc:1034] running on GCE node
I20260621 07:54:25.884196 27197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:25.884869 27197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:25.886047 27197 hybrid_clock.cc:648] HybridClock initialized: now 1782028465886025 us; error 40 us; skew 500 ppm
Jun 21 07:54:25 dist-test-slave-lbpt krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:25.888896 27197 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:54:25.889961 27197 webserver.cc:460] Webserver started at http://127.12.12.1:36367/ using document root <none> and password file <none>
I20260621 07:54:25.890609 27197 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:25.890677 27197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:25.894953 27197 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260621 07:54:25.897284 27246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:25.898713 27197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:54:25.898854 27197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "435bf73c81a140d885d3bd500094f182"
format_stamp: "Formatted at 2026-06-21 07:54:23 on dist-test-slave-lbpt"
server_key: "952c344175f7f5645a1fcf26bcf58682"
server_key_iv: "f2a7e03ae8be226d7ec9b7f09e2a3bc6"
server_key_version: "encryptionkey@0"
I20260621 07:54:25.899411 27197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:25.929675 27197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:25.933087 27197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:25.933301 27197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:25.933911 27197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:25.935065 27253 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 07:54:25.940459 27197 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 07:54:25.940538 27197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260621 07:54:25.940625 27197 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 07:54:25.942298 27197 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 07:54:25.942370 27197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260621 07:54:25.942747 27253 tablet_bootstrap.cc:492] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182: Bootstrap starting.
I20260621 07:54:25.953244 27197 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:45081
I20260621 07:54:25.953418 27360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:45081 every 8 connection(s)
I20260621 07:54:25.954479 27197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:54:25.958484 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 27197
I20260621 07:54:25.958894 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:38033
--local_ip_for_outbound_sockets=127.12.12.2
--tserver_master_addrs=127.12.12.62:46527
--webserver_port=33763
--webserver_interface=127.12.12.2
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:54:25.963241 27253 log.cc:824] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:25.970754 27253 tablet_bootstrap.cc:492] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:54:25.971158 27253 tablet_bootstrap.cc:492] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182: Bootstrap complete.
I20260621 07:54:25.972298 27253 ts_tablet_manager.cc:1397] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.004s
I20260621 07:54:25.974923 27253 raft_consensus.cc:348] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:25.975369 27253 raft_consensus.cc:729] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435bf73c81a140d885d3bd500094f182, State: Initialized, Role: FOLLOWER
I20260621 07:54:25.975867 27253 consensus_queue.cc:260] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:25.976755 27253 ts_tablet_manager.cc:1428] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20260621 07:54:26.025753 27363 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:54:25.956826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:45615)
0621 07:54:25.957430 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:25.957499 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:25.958750 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:26.024935 (+ 66185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":161,"threads_started":1}
W20260621 07:54:26.026405 27361 heartbeater.cc:644] Failed to heartbeat to 127.12.12.62:46527 (0 consecutive failures): Network error: Failed to ping master at 127.12.12.62:46527: Client connection negotiation failed: client connection to 127.12.12.62:46527: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 07:54:26.027253 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:26.027003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:52561)
0621 07:54:26.027130 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:26.027195 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 07:54:26.028012 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:26.027826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:33185)
0621 07:54:26.027925 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:26.027966 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 07:54:26.028198 27361 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:54:26.087962 27364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:26.088233 27364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:26.088305 27364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:26.092418 27364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:26.092482 27364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:26.092587 27364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:54:26.097898 27364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:38033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=33763
--tserver_master_addrs=127.12.12.62:46527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:26.098990 27364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:26.100028 27364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:26.110150 27372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:26.110354 27374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:26.110106 27371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:26.110736 27364 server_base.cc:1034] running on GCE node
I20260621 07:54:26.111333 27364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:26.112260 27364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:26.113443 27364 hybrid_clock.cc:648] HybridClock initialized: now 1782028466113411 us; error 53 us; skew 500 ppm
Jun 21 07:54:26 dist-test-slave-lbpt krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:26.116307 27364 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:54:26.117210 27364 webserver.cc:460] Webserver started at http://127.12.12.2:33763/ using document root <none> and password file <none>
I20260621 07:54:26.117739 27364 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:26.117798 27364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:26.121657 27364 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:54:26.123955 27381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:26.125270 27364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260621 07:54:26.125386 27364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "d5092a43d05643658a7da11156b33da9"
format_stamp: "Formatted at 2026-06-21 07:54:23 on dist-test-slave-lbpt"
server_key: "82a4c47d4c9c0a25f7afa5889f762a99"
server_key_iv: "ed4e0725d31c950ea170a13c117bf923"
server_key_version: "encryptionkey@0"
I20260621 07:54:26.125972 27364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:26.138765 27364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:26.141484 27364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:26.141636 27364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:26.142161 27364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:26.143239 27388 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 07:54:26.147261 27364 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 07:54:26.147318 27364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260621 07:54:26.147392 27364 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 07:54:26.149085 27364 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 07:54:26.149132 27364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260621 07:54:26.149389 27388 tablet_bootstrap.cc:492] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9: Bootstrap starting.
I20260621 07:54:26.155164 27388 log.cc:824] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:26.159693 27364 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:38033
I20260621 07:54:26.159750 27496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:38033 every 8 connection(s)
I20260621 07:54:26.160542 27364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:54:26.163210 27388 tablet_bootstrap.cc:492] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:54:26.163616 27388 tablet_bootstrap.cc:492] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9: Bootstrap complete.
I20260621 07:54:26.163920 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:26.162358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:52527)
0621 07:54:26.162934 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:26.163067 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":144,"threads_started":1}
I20260621 07:54:26.163964 27388 ts_tablet_manager.cc:1397] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
W20260621 07:54:26.164350 27497 heartbeater.cc:644] Failed to heartbeat to 127.12.12.62:46527 (0 consecutive failures): Network error: Failed to ping master at 127.12.12.62:46527: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
I20260621 07:54:26.164947 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:26.164717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:54355)
0621 07:54:26.164835 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:26.164888 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 07:54:26.165575 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:26.165350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:46501)
0621 07:54:26.165456 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:26.165505 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260621 07:54:26.165752 27497 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 07:54:26.167162 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 27364
I20260621 07:54:26.167600 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:36083
--local_ip_for_outbound_sockets=127.12.12.3
--tserver_master_addrs=127.12.12.62:46527
--webserver_port=38413
--webserver_interface=127.12.12.3
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:54:26.167243 27388 raft_consensus.cc:348] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:26.167796 27388 raft_consensus.cc:729] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5092a43d05643658a7da11156b33da9, State: Initialized, Role: FOLLOWER
I20260621 07:54:26.168293 27388 consensus_queue.cc:260] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:26.169348 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:26.169124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:42901)
0621 07:54:26.169242 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:26.169293 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 07:54:26.169384 27388 ts_tablet_manager.cc:1428] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260621 07:54:26.169998 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:26.169796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:43097)
0621 07:54:26.169902 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:26.169951 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 07:54:26.280422 27501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:26.280736 27501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:26.280817 27501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:26.284960 27501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:26.285054 27501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:26.285156 27501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:54:26.290452 27501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:36083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=38413
--tserver_master_addrs=127.12.12.62:46527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:26.291640 27501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:26.292716 27501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:26.302518 27507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:26.302547 27506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:26.302670 27501 server_base.cc:1034] running on GCE node
W20260621 07:54:26.302547 27509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:26.303295 27501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:26.304167 27501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:26.305356 27501 hybrid_clock.cc:648] HybridClock initialized: now 1782028466305320 us; error 51 us; skew 500 ppm
Jun 21 07:54:26 dist-test-slave-lbpt krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:26.308525 27501 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:54:26.309420 27501 webserver.cc:460] Webserver started at http://127.12.12.3:38413/ using document root <none> and password file <none>
I20260621 07:54:26.310016 27501 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:26.310110 27501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:26.314185 27501 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:54:26.316359 27516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:26.317790 27501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:54:26.317941 27501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "d01453ed24c64788a8db8af1b1e27947"
format_stamp: "Formatted at 2026-06-21 07:54:24 on dist-test-slave-lbpt"
server_key: "dac034e32b1454a52680901fe4ac823d"
server_key_iv: "e1af86d3c2ef7e2b7f88b009892a81c0"
server_key_version: "encryptionkey@0"
I20260621 07:54:26.318677 27501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:26.348258 27501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:26.351521 27501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:26.351688 27501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:26.352227 27501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:26.353029 27501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:54:26.353075 27501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:26.353114 27501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:54:26.353137 27501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:26.361930 27501 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:36083
I20260621 07:54:26.361999 27629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:36083 every 8 connection(s)
I20260621 07:54:26.363062 27501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:54:26.364776 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 27501
I20260621 07:54:26.367549 27633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:26.365601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:43005)
0621 07:54:26.366170 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:26.366287 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":139,"threads_started":1}
W20260621 07:54:26.368062 27630 heartbeater.cc:644] Failed to heartbeat to 127.12.12.62:46527 (0 consecutive failures): Network error: Failed to ping master at 127.12.12.62:46527: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
I20260621 07:54:26.368642 27633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:26.368426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:57537)
0621 07:54:26.368543 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:26.368590 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260621 07:54:26.369349 27633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:26.369121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:45961)
0621 07:54:26.369241 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:26.369290 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260621 07:54:26.369539 27630 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 07:54:27.029241 27634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:27.028780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:39025)
0621 07:54:27.029014 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:27.029102 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":75,"threads_started":1}
I20260621 07:54:27.171151 27635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:27.170782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:53111)
0621 07:54:27.171011 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:27.171066 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260621 07:54:27.370553 27636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:27.370079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:32781)
0621 07:54:27.370324 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:27.370379 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":83,"threads_started":1}
I20260621 07:54:27.771095 27637 raft_consensus.cc:482] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:54:27.771358 27637 raft_consensus.cc:504] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:27.772511 27637 leader_election.cc:288] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5092a43d05643658a7da11156b33da9 (127.12.12.2:38033), d01453ed24c64788a8db8af1b1e27947 (127.12.12.3:36083)
Jun 21 07:54:27 dist-test-slave-lbpt krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.2@KRBTEST.COM
Jun 21 07:54:27 dist-test-slave-lbpt krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.3@KRBTEST.COM
I20260621 07:54:27.786507 27638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:27.772786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:36083 (local address 127.12.12.1:37879)
0621 07:54:27.773209 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:27.773224 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:27.773335 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:27.774773 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:27.774776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:27.775031 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:27.775511 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:27.775527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:27.776764 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:27.776769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:27.777658 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:27.777667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:27.778377 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:27.778381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:27.778720 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:27.779510 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:27.779536 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:27.781721 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:27.784144 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:27.784147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:27.784148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:27.784335 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:27.784809 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:27.784811 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:27.784813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:27.784919 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:27.785525 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:27.785531 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:27.785771 (+ 240us) client_negotiation.cc:769] Sending connection context
0621 07:54:27.785950 (+ 179us) client_negotiation.cc:240] Negotiation successful
0621 07:54:27.786282 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":299,"spinlock_wait_cycles":49408,"thread_start_us":45,"threads_started":1}
I20260621 07:54:27.786530 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:27.772992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:42933 (local address 127.12.12.2:38033)
0621 07:54:27.773479 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:27.773491 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:27.773527 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:54:27.773784 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:27.773789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:27.774126 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:27.774451 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:27.775689 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:27.776624 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:27.777786 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:27.778249 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:27.778422 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:27.781531 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:27.781537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:27.781550 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:27.781576 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:27.783568 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:27.784474 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:27.784480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:27.784488 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:27.784641 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:27.785015 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:27.785020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:27.785021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:27.785407 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:27.785546 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:27.786068 (+ 522us) server_negotiation.cc:299] Negotiation successful
0621 07:54:27.786319 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":109,"threads_started":1}
I20260621 07:54:27.786554 27641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:27.773181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:37879 (local address 127.12.12.3:36083)
0621 07:54:27.773520 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:27.773539 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:27.773571 (+ 32us) server_negotiation.cc:407] Connection header received
0621 07:54:27.774211 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:27.774216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:27.774569 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:27.774828 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:27.775699 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:27.776641 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:27.777787 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:27.778256 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:27.778450 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:27.781924 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:27.781932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:27.781949 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:27.781984 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:27.784040 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:27.784473 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:27.784481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:27.784487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:27.784709 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:27.785017 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:27.785021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:27.785022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:27.785406 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:27.785547 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:27.786069 (+ 522us) server_negotiation.cc:299] Negotiation successful
0621 07:54:27.786345 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":63,"threads_started":1}
I20260621 07:54:27.786542 27639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:27.772786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:38033 (local address 127.12.12.1:42933)
0621 07:54:27.773110 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:27.773155 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:27.773279 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:27.774501 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:27.774507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:27.775019 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:27.775511 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:27.775526 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:27.776763 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:27.776767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:27.777660 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:27.777669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:27.778374 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:27.778377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:27.778627 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:27.779561 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:27.779576 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:27.781321 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:27.783693 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:27.783699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:27.783714 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:27.784049 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:27.784739 (+ 690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:27.784745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:27.784751 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:27.784915 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:27.785528 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:27.785536 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:27.785772 (+ 236us) client_negotiation.cc:769] Sending connection context
0621 07:54:27.785951 (+ 179us) client_negotiation.cc:240] Negotiation successful
0621 07:54:27.786362 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":260,"spinlock_wait_cycles":4352,"thread_start_us":84,"threads_started":1}
I20260621 07:54:27.788342 27450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8208d6302b194e57a4a6f661bc90e56a" candidate_uuid: "435bf73c81a140d885d3bd500094f182" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d5092a43d05643658a7da11156b33da9" is_pre_election: true
I20260621 07:54:27.788672 27450 raft_consensus.cc:2454] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 435bf73c81a140d885d3bd500094f182 in term 1.
W20260621 07:54:27.789453 27249 leader_election.cc:341] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d01453ed24c64788a8db8af1b1e27947 (127.12.12.3:36083): Not found: Tablet not found: 8208d6302b194e57a4a6f661bc90e56a
I20260621 07:54:27.789624 27249 leader_election.cc:302] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 435bf73c81a140d885d3bd500094f182, d5092a43d05643658a7da11156b33da9; no voters: d01453ed24c64788a8db8af1b1e27947
I20260621 07:54:27.789878 27637 raft_consensus.cc:2781] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260621 07:54:27.789960 27637 raft_consensus.cc:482] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:54:27.790005 27637 raft_consensus.cc:3037] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:54:27.792119 27637 raft_consensus.cc:504] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:27.792512 27637 leader_election.cc:288] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [CANDIDATE]: Term 2 election: Requested vote from peers d5092a43d05643658a7da11156b33da9 (127.12.12.2:38033), d01453ed24c64788a8db8af1b1e27947 (127.12.12.3:36083)
I20260621 07:54:27.793121 27450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8208d6302b194e57a4a6f661bc90e56a" candidate_uuid: "435bf73c81a140d885d3bd500094f182" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d5092a43d05643658a7da11156b33da9"
W20260621 07:54:27.793198 27249 leader_election.cc:341] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer d01453ed24c64788a8db8af1b1e27947 (127.12.12.3:36083): Not found: Tablet not found: 8208d6302b194e57a4a6f661bc90e56a
I20260621 07:54:27.793263 27450 raft_consensus.cc:3037] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:54:27.795257 27450 raft_consensus.cc:2454] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 435bf73c81a140d885d3bd500094f182 in term 2.
I20260621 07:54:27.795514 27248 leader_election.cc:302] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 435bf73c81a140d885d3bd500094f182, d5092a43d05643658a7da11156b33da9; no voters: d01453ed24c64788a8db8af1b1e27947
I20260621 07:54:27.795727 27637 raft_consensus.cc:2781] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 07:54:27.795900 27637 raft_consensus.cc:686] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [term 2 LEADER]: Becoming Leader. State: Replica: 435bf73c81a140d885d3bd500094f182, State: Running, Role: LEADER
I20260621 07:54:27.796238 27637 consensus_queue.cc:237] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:27.796626 27639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:27.796372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:57329)
0621 07:54:27.796470 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:27.796518 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260621 07:54:28.172391 27644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:28.171888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:52047)
0621 07:54:28.172232 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:28.172298 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20260621 07:54:28.371670 27645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:28.371301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:54227)
0621 07:54:28.371527 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:28.371582 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
I20260621 07:54:28.386397 27450 raft_consensus.cc:1261] T 8208d6302b194e57a4a6f661bc90e56a P d5092a43d05643658a7da11156b33da9 [term 2 FOLLOWER]: Refusing update from remote peer 435bf73c81a140d885d3bd500094f182: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260621 07:54:28.387239 27646 consensus_queue.cc:1035] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260621 07:54:28.389228 27644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:28.388956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:38879)
0621 07:54:28.389087 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:28.389150 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260621 07:54:28.392047 27249 consensus_peers.cc:487] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182 -> Peer d01453ed24c64788a8db8af1b1e27947 (127.12.12.3:36083): Couldn't send request to peer d01453ed24c64788a8db8af1b1e27947. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8208d6302b194e57a4a6f661bc90e56a. This is attempt 1: this message will repeat every 5th retry.
I20260621 07:54:28.392338 27644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:28.392048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:44087)
0621 07:54:28.392177 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:28.392263 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 07:54:28.392364 27653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:28.391564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:34975)
0621 07:54:28.392209 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:28.392278 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":119,"threads_started":1}
I20260621 07:54:28.939693 27654 ts_tablet_manager.cc:927] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: Initiating tablet copy from peer 435bf73c81a140d885d3bd500094f182 (127.12.12.1:45081)
I20260621 07:54:28.940349 27654 tablet_copy_client.cc:316] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: tablet copy: Beginning tablet copy session from remote peer at address 127.12.12.1:45081
Jun 21 07:54:28 dist-test-slave-lbpt krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.1@KRBTEST.COM
I20260621 07:54:28.951038 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:28.940952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:45081 (local address 127.12.12.3:53755)
0621 07:54:28.941276 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:28.941296 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:28.941419 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:28.941929 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:28.941935 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:28.942338 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:28.942529 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:28.942538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:28.943490 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:28.943492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:28.944152 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:28.944161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:28.944778 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:28.944781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:28.944858 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:28.945402 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:28.945416 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:28.947129 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:28.949164 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:28.949176 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:28.949182 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:28.949428 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:28.949740 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:28.949745 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:28.949747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:28.949835 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:28.950310 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:28.950317 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:28.950481 (+ 164us) client_negotiation.cc:769] Sending connection context
0621 07:54:28.950657 (+ 176us) client_negotiation.cc:240] Negotiation successful
0621 07:54:28.950841 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":127,"threads_started":1}
I20260621 07:54:28.951054 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:28.941136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:53755 (local address 127.12.12.1:45081)
0621 07:54:28.941427 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:28.941436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:28.941472 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:54:28.941529 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:28.941533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:28.941762 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:28.941888 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:28.942647 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:28.943383 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:28.944270 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:28.944658 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:28.944741 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:28.947273 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:28.947283 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:28.947295 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:28.947318 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:28.949033 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:28.949561 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:28.949567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:28.949568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:28.949647 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:28.949933 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:28.949936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:28.949938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:28.950197 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:28.950342 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:28.950756 (+ 414us) server_negotiation.cc:299] Negotiation successful
0621 07:54:28.950905 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":115,"threads_started":1}
I20260621 07:54:28.952214 27335 tablet_copy_service.cc:140] P 435bf73c81a140d885d3bd500094f182: Received BeginTabletCopySession request for tablet 8208d6302b194e57a4a6f661bc90e56a from peer d01453ed24c64788a8db8af1b1e27947 ({username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:53755)
I20260621 07:54:28.952395 27335 tablet_copy_service.cc:161] P 435bf73c81a140d885d3bd500094f182: Beginning new tablet copy session on tablet 8208d6302b194e57a4a6f661bc90e56a from peer d01453ed24c64788a8db8af1b1e27947 at {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:53755: session id = d01453ed24c64788a8db8af1b1e27947-8208d6302b194e57a4a6f661bc90e56a
I20260621 07:54:28.953639 27335 tablet_copy_source_session.cc:205] T 8208d6302b194e57a4a6f661bc90e56a P 435bf73c81a140d885d3bd500094f182: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 07:54:28.958030 27654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8208d6302b194e57a4a6f661bc90e56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:28.964614 27654 tablet_copy_client.cc:791] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: tablet copy: Starting download of 0 data blocks...
I20260621 07:54:28.964850 27654 tablet_copy_client.cc:655] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: tablet copy: Starting download of 1 WAL segments...
I20260621 07:54:28.966841 27654 tablet_copy_client.cc:531] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 07:54:28.968864 27654 tablet_bootstrap.cc:492] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: Bootstrap starting.
I20260621 07:54:28.974838 27654 log.cc:824] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:28.982066 27654 tablet_bootstrap.cc:492] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:54:28.982473 27654 tablet_bootstrap.cc:492] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: Bootstrap complete.
I20260621 07:54:28.982864 27654 ts_tablet_manager.cc:1397] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.004s
I20260621 07:54:28.985246 27654 raft_consensus.cc:348] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:28.985533 27654 raft_consensus.cc:729] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d01453ed24c64788a8db8af1b1e27947, State: Initialized, Role: FOLLOWER
I20260621 07:54:28.986019 27654 consensus_queue.cc:260] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5092a43d05643658a7da11156b33da9" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 38033 } } peers { permanent_uuid: "d01453ed24c64788a8db8af1b1e27947" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 36083 } } peers { permanent_uuid: "435bf73c81a140d885d3bd500094f182" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45081 } }
I20260621 07:54:28.987003 27654 ts_tablet_manager.cc:1428] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260621 07:54:28.987457 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:28.987143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:43431)
0621 07:54:28.987285 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:28.987362 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 07:54:28.987846 27335 tablet_copy_service.cc:342] P 435bf73c81a140d885d3bd500094f182: Request end of tablet copy session d01453ed24c64788a8db8af1b1e27947-8208d6302b194e57a4a6f661bc90e56a received from {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:53755
I20260621 07:54:28.987952 27335 tablet_copy_service.cc:433] P 435bf73c81a140d885d3bd500094f182: ending tablet copy session d01453ed24c64788a8db8af1b1e27947-8208d6302b194e57a4a6f661bc90e56a on tablet 8208d6302b194e57a4a6f661bc90e56a with peer d01453ed24c64788a8db8af1b1e27947
I20260621 07:54:28.988193 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:28.987903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:51121)
0621 07:54:28.988061 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:28.988124 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260621 07:54:29.393523 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:29.393080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:60415)
0621 07:54:29.393372 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:29.393429 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":115,"threads_started":1}
I20260621 07:54:29.393522 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:29.393100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:39639)
0621 07:54:29.393376 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:29.393430 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
I20260621 07:54:29.503156 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:29.502859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:35157)
0621 07:54:29.502991 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:29.503076 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260621 07:54:29.503306 27584 raft_consensus.cc:1203] T 8208d6302b194e57a4a6f661bc90e56a P d01453ed24c64788a8db8af1b1e27947 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260621 07:54:29.504765 27665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:29.504299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:44397)
0621 07:54:29.504600 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:29.504653 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":65,"threads_started":1}
I20260621 07:54:30.394945 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:30.394497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:55071)
0621 07:54:30.394784 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:30.394846 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":75,"threads_started":1}
I20260621 07:54:30.504346 27667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:30.503905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:51985)
0621 07:54:30.504188 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:30.504247 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
I20260621 07:54:30.505936 27668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:30.505543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:44761)
0621 07:54:30.505807 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:30.505857 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
I20260621 07:54:31.396572 27669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:31.396059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:48585)
0621 07:54:31.396371 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:31.396430 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":119,"threads_started":1}
I20260621 07:54:31.505722 27670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:31.505319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:38349)
0621 07:54:31.505563 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:31.505618 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260621 07:54:31.506541 27670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:31.506264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:54761)
0621 07:54:31.506386 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:31.506469 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 07:54:31.507205 27671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:31.506830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:33667)
0621 07:54:31.507080 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:31.507130 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
I20260621 07:54:32.397930 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:32.397459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:41473)
0621 07:54:32.397767 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:32.397829 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":132,"threads_started":1}
I20260621 07:54:32.507913 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:32.507385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:38423)
0621 07:54:32.507673 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:32.507776 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
I20260621 07:54:32.508432 27674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:32.507973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:49869)
0621 07:54:32.508247 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:32.508327 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":74,"threads_started":1}
I20260621 07:54:33.399298 27675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:33.398806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:39807)
0621 07:54:33.399136 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:33.399200 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":124,"threads_started":1}
I20260621 07:54:33.509178 27676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:33.508762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:33359)
0621 07:54:33.509018 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:33.509077 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
I20260621 07:54:33.509473 27677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:33.509074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:54311)
0621 07:54:33.509318 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:33.509377 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":88,"threads_started":1}
I20260621 07:54:34.400732 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:34.400208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:52419)
0621 07:54:34.400562 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:34.400630 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":94,"threads_started":1}
I20260621 07:54:34.510671 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:34.510098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:50913)
0621 07:54:34.510433 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:34.510520 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":83,"threads_started":1}
I20260621 07:54:34.510820 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:34.510285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:36453)
0621 07:54:34.510641 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:34.510696 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":124,"threads_started":1}
I20260621 07:54:35.402024 27681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:35.401586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:49275)
0621 07:54:35.401848 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:35.401922 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":82,"threads_started":1}
I20260621 07:54:35.511998 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:35.511517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:48613)
0621 07:54:35.511826 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:35.511900 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":95,"threads_started":1}
I20260621 07:54:35.512065 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:35.511592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:42095)
0621 07:54:35.511897 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:35.511973 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":134,"threads_started":1}
I20260621 07:54:36.403509 27685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:36.402994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:56375)
0621 07:54:36.403332 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:36.403408 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":103,"threads_started":1}
I20260621 07:54:36.513419 27686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:36.512979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:56887)
0621 07:54:36.513262 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:36.513324 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":125,"threads_started":1}
I20260621 07:54:36.513418 27687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:36.512979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:46351)
0621 07:54:36.513265 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:36.513325 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":105,"threads_started":1}
I20260621 07:54:37.404874 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:37.404405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:39131)
0621 07:54:37.404711 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:37.404776 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":119,"threads_started":1}
I20260621 07:54:37.514811 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:37.514300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:39803)
0621 07:54:37.514621 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:37.514694 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":114,"threads_started":1}
I20260621 07:54:37.514811 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:37.514300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:45521)
0621 07:54:37.514621 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:37.514694 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":114,"threads_started":1}
I20260621 07:54:38.406097 27691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:38.405669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:41437)
0621 07:54:38.405939 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:38.406000 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
I20260621 07:54:38.515988 27692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:38.515563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:56217)
0621 07:54:38.515837 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:38.515894 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":113,"threads_started":1}
I20260621 07:54:38.516021 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:38.515588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:36199)
0621 07:54:38.515888 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:38.515938 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":89,"threads_started":1}
I20260621 07:54:39.407445 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:39.406993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:38331)
0621 07:54:39.407288 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:39.407349 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":98,"threads_started":1}
I20260621 07:54:39.517342 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:39.516832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:37873)
0621 07:54:39.517084 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:39.517221 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260621 07:54:39.517342 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:39.516804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:48285)
0621 07:54:39.517171 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:39.517227 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":104,"threads_started":1}
I20260621 07:54:40.408684 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:40.408279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.2:37805)
0621 07:54:40.408529 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:40.408588 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
I20260621 07:54:40.518585 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:40.518104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.3:48879)
0621 07:54:40.518373 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:40.518484 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":105,"threads_started":1}
I20260621 07:54:40.518673 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:40.518202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:46527 (local address 127.12.12.1:37337)
0621 07:54:40.518462 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:40.518519 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
I20260621 07:54:41.375914 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 27197
I20260621 07:54:41.381762 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 27364
I20260621 07:54:41.387339 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 27501
2026-06-21T07:54:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19304 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:54:41 dist-test-slave-lbpt krb5kdc[27702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:54:41 dist-test-slave-lbpt krb5kdc[27702](info): set up 2 sockets
Jun 21 07:54:41 dist-test-slave-lbpt krb5kdc[27702](info): commencing operation
krb5kdc: starting...
W20260621 07:54:42.446707 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:54:42 dist-test-slave-lbpt krb5kdc[27702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:54:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:54:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:54:42.565276 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:33777
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:33777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:54:42.679100 27715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:42.679355 27715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:42.679394 27715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:42.683378 27715 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 07:54:42.683434 27715 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 07:54:42.683449 27715 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 07:54:42.683465 27715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:54:42.683482 27715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:42.683537 27715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:42.683570 27715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:54:42.683590 27715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:54:42.688795 27715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:33777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:33777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:42.689965 27715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:42.691047 27715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:42.699568 27723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:42.699667 27721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:42.699671 27720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:42.699751 27715 server_base.cc:1034] running on GCE node
I20260621 07:54:42.700565 27715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:42.701714 27715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:42.702907 27715 hybrid_clock.cc:648] HybridClock initialized: now 1782028482702886 us; error 38 us; skew 500 ppm
Jun 21 07:54:42 dist-test-slave-lbpt krb5kdc[27702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028482, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:42.705824 27715 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:54:42.706780 27715 webserver.cc:460] Webserver started at http://127.12.12.62:38481/ using document root <none> and password file <none>
I20260621 07:54:42.707322 27715 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:42.707382 27715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:42.707548 27715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:54:42.709645 27715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "adb63f18ce854712bea216af2d5e17fa"
format_stamp: "Formatted at 2026-06-21 07:54:42 on dist-test-slave-lbpt"
server_key: "bb2ff4e89eda4ab79c774f6657959b24"
server_key_iv: "70bc4b2dbb0373376c6e7ed9c122acdd"
server_key_version: "encryptionkey@0"
I20260621 07:54:42.710183 27715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "adb63f18ce854712bea216af2d5e17fa"
format_stamp: "Formatted at 2026-06-21 07:54:42 on dist-test-slave-lbpt"
server_key: "bb2ff4e89eda4ab79c774f6657959b24"
server_key_iv: "70bc4b2dbb0373376c6e7ed9c122acdd"
server_key_version: "encryptionkey@0"
I20260621 07:54:42.714052 27715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 07:54:42.716610 27730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:42.717852 27715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260621 07:54:42.717952 27715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "adb63f18ce854712bea216af2d5e17fa"
format_stamp: "Formatted at 2026-06-21 07:54:42 on dist-test-slave-lbpt"
server_key: "bb2ff4e89eda4ab79c774f6657959b24"
server_key_iv: "70bc4b2dbb0373376c6e7ed9c122acdd"
server_key_version: "encryptionkey@0"
I20260621 07:54:42.718041 27715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:42.736583 27715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:42.739715 27715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:42.739877 27715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:42.746856 27715 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:33777
I20260621 07:54:42.747162 27782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:33777 every 8 connection(s)
I20260621 07:54:42.747931 27715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:54:42.750777 27783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:42.751319 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 27715
I20260621 07:54:42.751447 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:54:42.751742 12336 external_mini_cluster.cc:1428] Setting key 9105dec2b4f0609db65d654c7dbfb10e
I20260621 07:54:42.757557 27783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: Bootstrap starting.
I20260621 07:54:42.760260 27783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:42.760989 27783 log.cc:824] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:54:42 dist-test-slave-lbpt krb5kdc[27702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:42.763312 27783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: No bootstrap required, opened a new log
I20260621 07:54:42.767414 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:42.753239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.12.12.62:33777)
0621 07:54:42.753696 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:42.753715 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:42.753760 (+ 45us) server_negotiation.cc:407] Connection header received
0621 07:54:42.754623 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:42.754648 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:42.755081 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:42.755446 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:42.756213 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:42.757700 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:42.758673 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:42.759353 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:42.759545 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:42.762110 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:42.762127 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:42.762144 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:42.762173 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:42.764235 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:42.764692 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:42.764698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:42.764704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:42.764843 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:42.765104 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:42.765107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:42.765109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:42.765673 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:42.765910 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:42.766096 (+ 186us) server_negotiation.cc:299] Negotiation successful
0621 07:54:42.766399 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":112,"threads_started":1}
I20260621 07:54:42.768000 27783 raft_consensus.cc:348] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } }
I20260621 07:54:42.768272 27783 raft_consensus.cc:374] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:42.768374 27783 raft_consensus.cc:729] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb63f18ce854712bea216af2d5e17fa, State: Initialized, Role: FOLLOWER
I20260621 07:54:42.769821 27783 consensus_queue.cc:260] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } }
I20260621 07:54:42.769953 27783 raft_consensus.cc:388] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:54:42.770036 27783 raft_consensus.cc:482] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:54:42.770139 27783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:42.771164 27783 raft_consensus.cc:504] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } }
I20260621 07:54:42.771517 27783 leader_election.cc:302] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adb63f18ce854712bea216af2d5e17fa; no voters:
I20260621 07:54:42.771858 27783 leader_election.cc:288] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:54:42.772019 27788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:54:42.772294 27788 raft_consensus.cc:686] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 1 LEADER]: Becoming Leader. State: Replica: adb63f18ce854712bea216af2d5e17fa, State: Running, Role: LEADER
I20260621 07:54:42.772598 27788 consensus_queue.cc:237] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } }
I20260621 07:54:42.773185 27783 sys_catalog.cc:564] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:54:42.774276 27790 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader adb63f18ce854712bea216af2d5e17fa. Latest consensus state: current_term: 1 leader_uuid: "adb63f18ce854712bea216af2d5e17fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } } }
I20260621 07:54:42.774397 27790 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [sys.catalog]: This master's current role is: LEADER
I20260621 07:54:42.774605 27789 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adb63f18ce854712bea216af2d5e17fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } } }
I20260621 07:54:42.774801 27789 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [sys.catalog]: This master's current role is: LEADER
W20260621 07:54:42.777695 27803 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:54:42.777783 27803 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:54:42.777860 27798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:54:42.778728 27798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:54:42.784615 27798 catalog_manager.cc:1347] Generated new cluster ID: 3bcd57c673b1419ea03a4d374ed19d4e
I20260621 07:54:42.784688 27798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:54:42.792999 27798 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:54:42.794258 27798 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:54:42.798960 27798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: Generated new TSK 0
I20260621 07:54:42.799978 27798 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:54:42.835424 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:33777
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:54:42.950184 27809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:42.950542 27809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:42.950649 27809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:42.954716 27809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:42.954811 27809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:42.954922 27809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:54:42.960214 27809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:33777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:42.961372 27809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:42.962453 27809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:42.972091 27817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:42.972097 27815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:42.972218 27809 server_base.cc:1034] running on GCE node
W20260621 07:54:42.972091 27814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:42.972741 27809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:42.973392 27809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:42.974583 27809 hybrid_clock.cc:648] HybridClock initialized: now 1782028482974549 us; error 51 us; skew 500 ppm
Jun 21 07:54:42 dist-test-slave-lbpt krb5kdc[27702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028482, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:42.977372 27809 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:54:42.978224 27809 webserver.cc:460] Webserver started at http://127.12.12.1:41661/ using document root <none> and password file <none>
I20260621 07:54:42.978847 27809 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:42.978933 27809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:42.979164 27809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:54:42.981171 27809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "1e3a50e2e445470cbbfd4464e0b2aed0"
format_stamp: "Formatted at 2026-06-21 07:54:42 on dist-test-slave-lbpt"
server_key: "cd809dc1abee0c1a7f6d7d04ccfe9a0b"
server_key_iv: "d60f3881849b607b619cf8c76cfe411f"
server_key_version: "encryptionkey@0"
I20260621 07:54:42.981729 27809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "1e3a50e2e445470cbbfd4464e0b2aed0"
format_stamp: "Formatted at 2026-06-21 07:54:42 on dist-test-slave-lbpt"
server_key: "cd809dc1abee0c1a7f6d7d04ccfe9a0b"
server_key_iv: "d60f3881849b607b619cf8c76cfe411f"
server_key_version: "encryptionkey@0"
I20260621 07:54:42.985399 27809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260621 07:54:42.987908 27824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:42.989140 27809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:54:42.989279 27809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "1e3a50e2e445470cbbfd4464e0b2aed0"
format_stamp: "Formatted at 2026-06-21 07:54:42 on dist-test-slave-lbpt"
server_key: "cd809dc1abee0c1a7f6d7d04ccfe9a0b"
server_key_iv: "d60f3881849b607b619cf8c76cfe411f"
server_key_version: "encryptionkey@0"
I20260621 07:54:42.989392 27809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:43.007952 27809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:43.011510 27809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:43.011729 27809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:43.012342 27809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:43.013226 27809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:54:43.013289 27809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:43.013357 27809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:54:43.013403 27809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:43.023115 27809 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:36323
I20260621 07:54:43.023171 27937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:36323 every 8 connection(s)
I20260621 07:54:43.024031 27809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:54:43.032284 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 27809
I20260621 07:54:43.032392 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:54:43.032713 12336 external_mini_cluster.cc:1428] Setting key e7aab7eb81c426305547572ee6d4b021
Jun 21 07:54:43 dist-test-slave-lbpt krb5kdc[27702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028482, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:43.037909 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.025841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:48089 (local address 127.12.12.62:33777)
0621 07:54:43.025971 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:43.025974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:43.027012 (+ 1038us) server_negotiation.cc:407] Connection header received
0621 07:54:43.027927 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:43.027931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:43.027991 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:43.028090 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:43.029261 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.029812 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.031051 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.031293 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.031404 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.034151 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:43.034155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:43.034157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:43.034188 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:43.035844 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:43.036366 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:43.036370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:43.036372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:43.036451 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:43.036805 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:43.036809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:43.036810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:43.036986 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:43.037092 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:43.037611 (+ 519us) server_negotiation.cc:299] Negotiation successful
0621 07:54:43.037738 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:54:43.038767 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.026195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:48089)
0621 07:54:43.026868 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:43.026909 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:43.027724 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:43.028280 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:43.028287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:43.028645 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:43.029115 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:43.029128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.029935 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.029943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:43.030927 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:43.030936 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.031444 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.031449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:43.031613 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.032208 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:43.032224 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:43.033979 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:43.035964 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:43.035970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:43.035986 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:43.036258 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:43.036538 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:43.036542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:43.036544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:43.036704 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:43.037090 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:43.037098 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:43.037351 (+ 253us) client_negotiation.cc:769] Sending connection context
0621 07:54:43.037536 (+ 185us) client_negotiation.cc:240] Negotiation successful
0621 07:54:43.037865 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":162,"threads_started":1}
I20260621 07:54:43.040077 27938 heartbeater.cc:347] Connected to a master server at 127.12.12.62:33777
I20260621 07:54:43.040323 27938 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:43.040795 27938 heartbeater.cc:511] Master 127.12.12.62:33777 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
I20260621 07:54:43.042269 27747 ts_manager.cc:194] Registered new tserver with Master: 1e3a50e2e445470cbbfd4464e0b2aed0 (127.12.12.1:36323)
I20260621 07:54:43.043421 27747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:48089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:54:43.059171 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:33777
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:54:43.173393 27943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:43.173789 27943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:43.173877 27943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:43.178646 27943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:43.178702 27943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:43.178782 27943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:54:43.184131 27943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:33777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:43.185232 27943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:43.186247 27943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:43.196156 27951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:43.196182 27949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:43.196225 27943 server_base.cc:1034] running on GCE node
W20260621 07:54:43.196177 27948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:43.196718 27943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:43.197340 27943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:43.198482 27943 hybrid_clock.cc:648] HybridClock initialized: now 1782028483198463 us; error 32 us; skew 500 ppm
Jun 21 07:54:43 dist-test-slave-lbpt krb5kdc[27702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:43.201148 27943 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:54:43.202064 27943 webserver.cc:460] Webserver started at http://127.12.12.2:45741/ using document root <none> and password file <none>
I20260621 07:54:43.202680 27943 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:43.202744 27943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:43.202988 27943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:54:43.204985 27943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "6f483172cbbc4357adf57e8db6356b12"
format_stamp: "Formatted at 2026-06-21 07:54:43 on dist-test-slave-lbpt"
server_key: "6895cb5b30dd8949d968643afcda516a"
server_key_iv: "456c9427d7bf4b066093395a5f94d1f3"
server_key_version: "encryptionkey@0"
I20260621 07:54:43.205575 27943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "6f483172cbbc4357adf57e8db6356b12"
format_stamp: "Formatted at 2026-06-21 07:54:43 on dist-test-slave-lbpt"
server_key: "6895cb5b30dd8949d968643afcda516a"
server_key_iv: "456c9427d7bf4b066093395a5f94d1f3"
server_key_version: "encryptionkey@0"
I20260621 07:54:43.209435 27943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260621 07:54:43.212164 27958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:43.213454 27943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:54:43.213579 27943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "6f483172cbbc4357adf57e8db6356b12"
format_stamp: "Formatted at 2026-06-21 07:54:43 on dist-test-slave-lbpt"
server_key: "6895cb5b30dd8949d968643afcda516a"
server_key_iv: "456c9427d7bf4b066093395a5f94d1f3"
server_key_version: "encryptionkey@0"
I20260621 07:54:43.213725 27943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:43.230175 27943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:43.233566 27943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:43.233793 27943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:43.234480 27943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:43.235361 27943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:54:43.235409 27943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:43.235476 27943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:54:43.235510 27943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:43.244570 27943 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:45059
I20260621 07:54:43.244590 28071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:45059 every 8 connection(s)
I20260621 07:54:43.245528 27943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:54:43.246033 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 27943
I20260621 07:54:43.246170 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:54:43.246511 12336 external_mini_cluster.cc:1428] Setting key 42bfe1711af7a363f3424e10d6f07b40
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
Jun 21 07:54:43 dist-test-slave-lbpt krb5kdc[27702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:43.264200 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.247484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:50867 (local address 127.12.12.62:33777)
0621 07:54:43.247927 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:43.247932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:43.248951 (+ 1019us) server_negotiation.cc:407] Connection header received
0621 07:54:43.250085 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:43.250087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:43.250137 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:43.250216 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:43.251726 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.252705 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.253956 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.254350 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.254494 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.259853 (+ 5359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:43.259860 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:43.259866 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:43.259906 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:43.261485 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:43.262235 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:43.262238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:43.262240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:43.262360 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:43.262910 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:43.262914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:43.262916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:43.263122 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:43.263203 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:43.263946 (+ 743us) server_negotiation.cc:299] Negotiation successful
0621 07:54:43.264073 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337}
I20260621 07:54:43.265528 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.248009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:50867)
0621 07:54:43.248778 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:43.248834 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:43.249826 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:43.250472 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:43.250484 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:43.251056 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:43.251561 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:43.251574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.252856 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.252859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:43.253822 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:43.253832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.254572 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.254575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:43.254739 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.255510 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:43.255531 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:43.259631 (+ 4100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:43.261615 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:43.261625 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:43.261646 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:43.262065 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:43.262500 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:43.262506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:43.262511 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:43.262725 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:43.263232 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:43.263242 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:43.263624 (+ 382us) client_negotiation.cc:769] Sending connection context
0621 07:54:43.263885 (+ 261us) client_negotiation.cc:240] Negotiation successful
0621 07:54:43.264322 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":151,"threads_started":1}
I20260621 07:54:43.267053 28072 heartbeater.cc:347] Connected to a master server at 127.12.12.62:33777
I20260621 07:54:43.267323 28072 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:43.267889 28072 heartbeater.cc:511] Master 127.12.12.62:33777 requested a full tablet report, sending...
I20260621 07:54:43.269090 27747 ts_manager.cc:194] Registered new tserver with Master: 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059)
I20260621 07:54:43.269886 27747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:50867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:54:43.275439 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:33777
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:54:43.385766 28077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:54:43.386071 28077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:54:43.386143 28077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:54:43.390198 28077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:54:43.390291 28077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:54:43.390388 28077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:54:43.395962 28077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:33777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:54:43.397238 28077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:54:43.398399 28077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:54:43.407927 28082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:43.407927 28085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:54:43.408061 28083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:54:43.408420 28077 server_base.cc:1034] running on GCE node
I20260621 07:54:43.408748 28077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:54:43.409322 28077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:54:43.410472 28077 hybrid_clock.cc:648] HybridClock initialized: now 1782028483410418 us; error 60 us; skew 500 ppm
Jun 21 07:54:43 dist-test-slave-lbpt krb5kdc[27702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:54:43.413153 28077 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:54:43.413982 28077 webserver.cc:460] Webserver started at http://127.12.12.3:32983/ using document root <none> and password file <none>
I20260621 07:54:43.414588 28077 fs_manager.cc:359] Metadata directory not provided
I20260621 07:54:43.414677 28077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:54:43.414899 28077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:54:43.416920 28077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7"
format_stamp: "Formatted at 2026-06-21 07:54:43 on dist-test-slave-lbpt"
server_key: "63d984fc73959202d9052d2f7cd6ab66"
server_key_iv: "ee8e5af0eb7411ab15518705762a18d5"
server_key_version: "encryptionkey@0"
I20260621 07:54:43.417470 28077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7"
format_stamp: "Formatted at 2026-06-21 07:54:43 on dist-test-slave-lbpt"
server_key: "63d984fc73959202d9052d2f7cd6ab66"
server_key_iv: "ee8e5af0eb7411ab15518705762a18d5"
server_key_version: "encryptionkey@0"
I20260621 07:54:43.421219 28077 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260621 07:54:43.423964 28092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:43.425225 28077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:54:43.425328 28077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7"
format_stamp: "Formatted at 2026-06-21 07:54:43 on dist-test-slave-lbpt"
server_key: "63d984fc73959202d9052d2f7cd6ab66"
server_key_iv: "ee8e5af0eb7411ab15518705762a18d5"
server_key_version: "encryptionkey@0"
I20260621 07:54:43.425415 28077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:54:43.451411 28077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:54:43.454324 28077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:54:43.454545 28077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:54:43.455113 28077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:54:43.455873 28077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:54:43.455919 28077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:43.455958 28077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:54:43.455979 28077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:54:43.464942 28077 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:34081
I20260621 07:54:43.464964 28205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:34081 every 8 connection(s)
I20260621 07:54:43.465952 28077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:54:43.471963 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 28077
I20260621 07:54:43.472095 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:54:43.472566 12336 external_mini_cluster.cc:1428] Setting key 49f3aed659bfb828f32f070556fc814c
Jun 21 07:54:43 dist-test-slave-lbpt krb5kdc[27702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:54:43.480396 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.467859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:53519 (local address 127.12.12.62:33777)
0621 07:54:43.468008 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:43.468011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:43.469052 (+ 1041us) server_negotiation.cc:407] Connection header received
0621 07:54:43.469932 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:43.469935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:43.469983 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:43.470075 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:43.471388 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.472272 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.473750 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.474092 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.474220 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.476916 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:43.476920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:43.476923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:43.476951 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:43.478260 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:43.478829 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:43.478834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:43.478836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:43.478924 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:43.479279 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:43.479283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:43.479284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:43.479480 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:43.479585 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:43.480100 (+ 515us) server_negotiation.cc:299] Negotiation successful
0621 07:54:43.480231 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:54:43.481259 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.468249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:53519)
0621 07:54:43.468900 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:43.468940 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:43.469715 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:43.470259 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:43.470266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:43.470734 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:54:43.471228 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:43.471242 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.472400 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.472403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:43.473591 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:43.473640 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.474262 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.474267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:43.474509 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.475225 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:54:43.475241 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:54:43.476754 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:54:43.478377 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:43.478384 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:43.478400 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:43.478722 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:43.479014 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:54:43.479019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:54:43.479021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:54:43.479177 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:54:43.479583 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:54:43.479590 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:54:43.479856 (+ 266us) client_negotiation.cc:769] Sending connection context
0621 07:54:43.480042 (+ 186us) client_negotiation.cc:240] Negotiation successful
0621 07:54:43.480397 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":118,"threads_started":1}
I20260621 07:54:43.482478 28206 heartbeater.cc:347] Connected to a master server at 127.12.12.62:33777
I20260621 07:54:43.482750 28206 heartbeater.cc:464] Registering TS with master...
I20260621 07:54:43.483271 28206 heartbeater.cc:511] Master 127.12.12.62:33777 requested a full tablet report, sending...
I20260621 07:54:43.484424 27747 ts_manager.cc:194] Registered new tserver with Master: c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081)
I20260621 07:54:43.485028 27747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:53519
I20260621 07:54:43.486979 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:54:43.496728 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.489092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.12.12.62:33777)
0621 07:54:43.489210 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:43.489213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:43.489282 (+ 69us) server_negotiation.cc:407] Connection header received
0621 07:54:43.489378 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:43.489380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:43.489416 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:43.489488 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:54:43.490216 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.490825 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.491784 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.492001 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.492117 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.493577 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:54:43.493580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:54:43.493583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:54:43.493605 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:54:43.495169 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:43.495564 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:43.495568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:43.495570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:43.495650 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:54:43.495875 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:54:43.495877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:54:43.495878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:54:43.496108 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:54:43.496209 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:43.496473 (+ 264us) server_negotiation.cc:299] Negotiation successful
0621 07:54:43.496600 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:54:43.499779 27747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:54:43.502820 27747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:54:43.517603 28217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.511897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:45059 (local address 127.0.0.1:52466)
0621 07:54:43.512467 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:43.512482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:43.512563 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:43.513301 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:43.513305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:43.513332 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:43.513501 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:43.513508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.514665 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.514702 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:43.516007 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:43.516017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.516965 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.516969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:43.517091 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.517138 (+ 47us) client_negotiation.cc:769] Sending connection context
0621 07:54:43.517262 (+ 124us) client_negotiation.cc:240] Negotiation successful
0621 07:54:43.517437 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":165,"spinlock_wait_cycles":1408,"thread_start_us":94,"threads_started":1}
I20260621 07:54:43.517606 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.512053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.12.12.2:45059)
0621 07:54:43.512802 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:43.512817 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:43.512848 (+ 31us) server_negotiation.cc:407] Connection header received
0621 07:54:43.512915 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:43.512918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:43.513133 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:43.513287 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:43.513733 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.514533 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.516134 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.516809 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.516923 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.517081 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:43.517310 (+ 229us) server_negotiation.cc:299] Negotiation successful
0621 07:54:43.517412 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":157,"threads_started":1}
I20260621 07:54:43.518244 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.511364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:36323 (local address 127.0.0.1:54624)
0621 07:54:43.512330 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:43.512344 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:43.512424 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:43.513307 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:43.513309 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:43.513341 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:43.513525 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:43.513532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.515032 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.515038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:43.516462 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:43.516473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.517729 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.517734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:43.517864 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.517885 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:54:43.517967 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 07:54:43.518036 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":121,"spinlock_wait_cycles":5376,"thread_start_us":104,"threads_started":1}
I20260621 07:54:43.518514 28220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.511856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.12.12.1:36323)
0621 07:54:43.512114 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:43.512134 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:43.512566 (+ 432us) server_negotiation.cc:407] Connection header received
0621 07:54:43.512639 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:43.512643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:43.512840 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:43.512995 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:43.513772 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.514887 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.516603 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.517428 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.517523 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.517662 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:43.518248 (+ 586us) server_negotiation.cc:299] Negotiation successful
0621 07:54:43.518347 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
I20260621 07:54:43.518518 28219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.511354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.12.12.3:34081)
0621 07:54:43.512290 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:43.512301 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:43.512328 (+ 27us) server_negotiation.cc:407] Connection header received
0621 07:54:43.512617 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:43.512623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:43.512840 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:43.512995 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:43.513301 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.514546 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.516593 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.517457 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.517578 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.517742 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:43.518268 (+ 526us) server_negotiation.cc:299] Negotiation successful
0621 07:54:43.518355 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":106,"threads_started":1}
I20260621 07:54:43.518815 28218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.511181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:34081 (local address 127.0.0.1:57244)
0621 07:54:43.511720 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:43.511774 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:43.512424 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:43.513010 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:43.513016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:43.513040 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:43.513180 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:43.513192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.514703 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.514706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:43.516462 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:43.516473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.517811 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.517818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:43.517957 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.517983 (+ 26us) client_negotiation.cc:769] Sending connection context
0621 07:54:43.518057 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 07:54:43.518140 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":258,"thread_start_us":74,"threads_started":1}
I20260621 07:54:43.521340 28140 tablet_service.cc:1467] Processing CreateTablet for tablet c4e85e68992044458b6eaef0f4587380 (DEFAULT_TABLE table=test-table [id=49f2960b9f5248049c078086e2446368]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:54:43.521457 27872 tablet_service.cc:1467] Processing CreateTablet for tablet c4e85e68992044458b6eaef0f4587380 (DEFAULT_TABLE table=test-table [id=49f2960b9f5248049c078086e2446368]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:54:43.521456 28006 tablet_service.cc:1467] Processing CreateTablet for tablet c4e85e68992044458b6eaef0f4587380 (DEFAULT_TABLE table=test-table [id=49f2960b9f5248049c078086e2446368]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:54:43.522462 28140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4e85e68992044458b6eaef0f4587380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:43.522490 28006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4e85e68992044458b6eaef0f4587380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:43.522490 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4e85e68992044458b6eaef0f4587380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:54:43.529026 28223 tablet_bootstrap.cc:492] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7: Bootstrap starting.
I20260621 07:54:43.530453 28225 tablet_bootstrap.cc:492] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: Bootstrap starting.
I20260621 07:54:43.530761 28224 tablet_bootstrap.cc:492] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0: Bootstrap starting.
I20260621 07:54:43.531796 28223 tablet_bootstrap.cc:654] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:43.532555 28223 log.cc:824] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:43.532644 28224 tablet_bootstrap.cc:654] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:43.533056 28225 tablet_bootstrap.cc:654] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: Neither blocks nor log segments found. Creating new log.
I20260621 07:54:43.533965 28225 log.cc:824] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:43.533965 28224 log.cc:824] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0: Log is configured to *not* fsync() on all Append() calls
I20260621 07:54:43.534924 28223 tablet_bootstrap.cc:492] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7: No bootstrap required, opened a new log
I20260621 07:54:43.535141 28223 ts_tablet_manager.cc:1397] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:54:43.535979 28225 tablet_bootstrap.cc:492] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: No bootstrap required, opened a new log
I20260621 07:54:43.536192 28225 ts_tablet_manager.cc:1397] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:54:43.536289 28224 tablet_bootstrap.cc:492] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0: No bootstrap required, opened a new log
I20260621 07:54:43.536458 28224 ts_tablet_manager.cc:1397] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:54:43.538292 28223 raft_consensus.cc:348] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } }
I20260621 07:54:43.538640 28223 raft_consensus.cc:374] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:43.538724 28223 raft_consensus.cc:729] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b655ed1d1d4ffab5e07dd085a2dda7, State: Initialized, Role: FOLLOWER
I20260621 07:54:43.539209 28223 consensus_queue.cc:260] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } }
I20260621 07:54:43.540277 28223 ts_tablet_manager.cc:1428] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:54:43.540195 28224 raft_consensus.cc:348] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } }
I20260621 07:54:43.540202 28225 raft_consensus.cc:348] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } }
I20260621 07:54:43.540457 28225 raft_consensus.cc:374] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:43.540535 28224 raft_consensus.cc:374] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:54:43.540557 28225 raft_consensus.cc:729] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f483172cbbc4357adf57e8db6356b12, State: Initialized, Role: FOLLOWER
I20260621 07:54:43.540602 28224 raft_consensus.cc:729] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e3a50e2e445470cbbfd4464e0b2aed0, State: Initialized, Role: FOLLOWER
I20260621 07:54:43.540982 28206 heartbeater.cc:503] Master 127.12.12.62:33777 was elected leader, sending a full tablet report...
I20260621 07:54:43.541034 28225 consensus_queue.cc:260] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } }
I20260621 07:54:43.541034 28224 consensus_queue.cc:260] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } }
I20260621 07:54:43.541644 27938 heartbeater.cc:503] Master 127.12.12.62:33777 was elected leader, sending a full tablet report...
I20260621 07:54:43.541982 28225 ts_tablet_manager.cc:1428] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260621 07:54:43.541982 28224 ts_tablet_manager.cc:1428] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260621 07:54:43.542294 28072 heartbeater.cc:503] Master 127.12.12.62:33777 was elected leader, sending a full tablet report...
I20260621 07:54:43.676381 28229 raft_consensus.cc:482] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:54:43.676628 28229 raft_consensus.cc:504] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } }
I20260621 07:54:43.677963 28229 leader_election.cc:288] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e3a50e2e445470cbbfd4464e0b2aed0 (127.12.12.1:36323), 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059)
I20260621 07:54:43.681874 28220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.678313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:60157 (local address 127.12.12.1:36323)
0621 07:54:43.678492 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:43.678497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:43.678519 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:54:43.678594 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:43.678600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:43.678656 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:43.678770 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:43.679157 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.679798 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.680862 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.681291 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.681375 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.681464 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:43.681659 (+ 195us) server_negotiation.cc:299] Negotiation successful
0621 07:54:43.681739 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:54:43.681887 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.678511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:56279 (local address 127.12.12.2:45059)
0621 07:54:43.678636 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:43.678640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:43.678900 (+ 260us) server_negotiation.cc:407] Connection header received
0621 07:54:43.678988 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:43.678991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:43.679045 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:43.679131 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:54:43.679394 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.679897 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.680920 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.681315 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.681437 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.681499 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:43.681705 (+ 206us) server_negotiation.cc:299] Negotiation successful
0621 07:54:43.681781 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:54:43.681874 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.678191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:36323 (local address 127.12.12.3:60157)
0621 07:54:43.678330 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:43.678348 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:43.678469 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:43.678817 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:43.678820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:43.678839 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:43.679011 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:43.679018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.679912 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.679918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:43.680732 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:43.680741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.681418 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.681422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:43.681532 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.681558 (+ 26us) client_negotiation.cc:769] Sending connection context
0621 07:54:43.681618 (+ 60us) client_negotiation.cc:240] Negotiation successful
0621 07:54:43.681720 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260621 07:54:43.681883 28232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.678280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:45059 (local address 127.12.12.3:56279)
0621 07:54:43.678781 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:43.678797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:54:43.678894 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:54:43.679152 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:54:43.679155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:54:43.679172 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:54:43.679286 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:43.679292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.680000 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.680003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:43.680773 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:54:43.680781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.681459 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.681463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:54:43.681562 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.681582 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 07:54:43.681674 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 07:54:43.681737 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":152,"threads_started":1}
I20260621 07:54:43.682610 28026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e85e68992044458b6eaef0f4587380" candidate_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f483172cbbc4357adf57e8db6356b12" is_pre_election: true
I20260621 07:54:43.682659 27892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e85e68992044458b6eaef0f4587380" candidate_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" is_pre_election: true
I20260621 07:54:43.682931 27892 raft_consensus.cc:2454] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1b655ed1d1d4ffab5e07dd085a2dda7 in term 0.
I20260621 07:54:43.682936 28026 raft_consensus.cc:2454] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1b655ed1d1d4ffab5e07dd085a2dda7 in term 0.
I20260621 07:54:43.683418 28096 leader_election.cc:302] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f483172cbbc4357adf57e8db6356b12, c1b655ed1d1d4ffab5e07dd085a2dda7; no voters:
I20260621 07:54:43.683709 28229 raft_consensus.cc:2781] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:54:43.683794 28229 raft_consensus.cc:482] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:54:43.683845 28229 raft_consensus.cc:3037] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:43.684865 28229 raft_consensus.cc:504] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } }
I20260621 07:54:43.685199 28229 leader_election.cc:288] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [CANDIDATE]: Term 1 election: Requested vote from peers 1e3a50e2e445470cbbfd4464e0b2aed0 (127.12.12.1:36323), 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059)
I20260621 07:54:43.685514 28026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e85e68992044458b6eaef0f4587380" candidate_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f483172cbbc4357adf57e8db6356b12"
I20260621 07:54:43.685513 27892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e85e68992044458b6eaef0f4587380" candidate_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0"
I20260621 07:54:43.685642 27892 raft_consensus.cc:3037] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:43.685642 28026 raft_consensus.cc:3037] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:54:43.686985 27892 raft_consensus.cc:2454] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b655ed1d1d4ffab5e07dd085a2dda7 in term 1.
I20260621 07:54:43.686985 28026 raft_consensus.cc:2454] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b655ed1d1d4ffab5e07dd085a2dda7 in term 1.
I20260621 07:54:43.687297 28096 leader_election.cc:302] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f483172cbbc4357adf57e8db6356b12, c1b655ed1d1d4ffab5e07dd085a2dda7; no voters:
I20260621 07:54:43.687458 28229 raft_consensus.cc:2781] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:54:43.687705 28229 raft_consensus.cc:686] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 1 LEADER]: Becoming Leader. State: Replica: c1b655ed1d1d4ffab5e07dd085a2dda7, State: Running, Role: LEADER
I20260621 07:54:43.688083 28229 consensus_queue.cc:237] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } }
I20260621 07:54:43.691416 27745 catalog_manager.cc:5495] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3). New cstate: current_term: 1 leader_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } health_report { overall_health: UNKNOWN } } }
I20260621 07:54:43.707876 28219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.703763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.12.12.3:34081)
0621 07:54:43.703935 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 07:54:43.703939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:54:43.703954 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:54:43.704116 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:54:43.704118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:54:43.704177 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:54:43.704296 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:54:43.704560 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.705093 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.705984 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:54:43.706234 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:54:43.706329 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:54:43.706905 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:54:43.707374 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:54:43.707478 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 07:54:43.707589 (+ 111us) server_negotiation.cc:299] Negotiation successful
0621 07:54:43.707655 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:54:43.714249 28026 raft_consensus.cc:1261] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12 [term 1 FOLLOWER]: Refusing update from remote peer c1b655ed1d1d4ffab5e07dd085a2dda7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:54:43.714249 27892 raft_consensus.cc:1261] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Refusing update from remote peer c1b655ed1d1d4ffab5e07dd085a2dda7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:54:43.714994 28229 consensus_queue.cc:1035] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:54:43.715190 28233 consensus_queue.cc:1035] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 07:54:43.716938 28207 tablet.cc:2369] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:54:43.723577 28238 mvcc.cc:205] Tried to move back new op lower bound from 7299188669286182912 to 7299188669189316608. Current Snapshot: MvccSnapshot[applied={T|T < 7299188669286182912}]
I20260621 07:54:43.726388 28237 mvcc.cc:205] Tried to move back new op lower bound from 7299188669286182912 to 7299188669189316608. Current Snapshot: MvccSnapshot[applied={T|T < 7299188669286182912}]
I20260621 07:54:43.733067 27747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:54:43.733289 27747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:54:43.736002 27747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: Sending DeleteTablet for 3 replicas of tablet c4e85e68992044458b6eaef0f4587380
I20260621 07:54:43.736901 28140 tablet_service.cc:1514] Processing DeleteTablet for tablet c4e85e68992044458b6eaef0f4587380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:54:43 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:57244
I20260621 07:54:43.737179 28006 tablet_service.cc:1514] Processing DeleteTablet for tablet c4e85e68992044458b6eaef0f4587380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:54:43 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:52466
I20260621 07:54:43.737293 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 27809
I20260621 07:54:43.737471 28245 tablet_replica.cc:330] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7: stopping tablet replica
I20260621 07:54:43.737792 28245 raft_consensus.cc:2229] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:54:43.738161 28245 raft_consensus.cc:2258] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:54:43.739825 28245 ts_tablet_manager.cc:1905] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:54:43.742771 28245 ts_tablet_manager.cc:1918] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:54:43.742961 28245 log.cc:1196] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/c4e85e68992044458b6eaef0f4587380
I20260621 07:54:43.742951 28246 tablet_replica.cc:330] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: stopping tablet replica
I20260621 07:54:43.743326 28246 raft_consensus.cc:2229] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:54:43.743415 28245 ts_tablet_manager.cc:1939] T c4e85e68992044458b6eaef0f4587380 P c1b655ed1d1d4ffab5e07dd085a2dda7: Deleting consensus metadata
I20260621 07:54:43.743841 28246 raft_consensus.cc:2258] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260621 07:54:43.744385 27731 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.1:36323 (error 108)
I20260621 07:54:43.745075 27732 catalog_manager.cc:4841] TS c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081): tablet c4e85e68992044458b6eaef0f4587380 (table test-table [id=49f2960b9f5248049c078086e2446368]) successfully deleted
I20260621 07:54:43.745366 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 27943
I20260621 07:54:43.745795 28246 ts_tablet_manager.cc:1905] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:54:43.751356 28217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.751052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:36323 (local address 127.0.0.1:54636)
0621 07:54:43.751196 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:43.751272 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:36323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260621 07:54:43.751543 27731 catalog_manager.cc:4584] TS 1e3a50e2e445470cbbfd4464e0b2aed0 (127.12.12.1:36323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4e85e68992044458b6eaef0f4587380: Network error: Client connection negotiation failed: client connection to 127.12.12.1:36323: connect: Connection refused (error 111)
I20260621 07:54:43.753067 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 28077
I20260621 07:54:43.753180 28217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:54:43.752906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:45059 (local address 127.0.0.1:52468)
0621 07:54:43.753054 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0621 07:54:43.753100 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.2:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260621 07:54:43.758742 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 27715
I20260621 07:55:03.764869 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:33777
--webserver_interface=127.12.12.62
--webserver_port=38481
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:33777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:03.876071 28248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:03.876417 28248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:03.876508 28248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:03.880574 28248 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 07:55:03.880676 28248 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 07:55:03.880704 28248 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 07:55:03.880738 28248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:55:03.880779 28248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:03.880818 28248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:03.880849 28248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:55:03.880887 28248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:55:03.886405 28248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:33777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:33777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=38481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:03.887638 28248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:03.888762 28248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:03.902558 28255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:03.902522 28257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:03.902685 28248 server_base.cc:1034] running on GCE node
W20260621 07:55:03.902526 28254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:03.903512 28248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:03.904736 28248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:03.905944 28248 hybrid_clock.cc:648] HybridClock initialized: now 1782028503905924 us; error 36 us; skew 500 ppm
Jun 21 07:55:03 dist-test-slave-lbpt krb5kdc[27702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028503, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:03.909245 28248 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:55:03.910526 28248 webserver.cc:460] Webserver started at http://127.12.12.62:38481/ using document root <none> and password file <none>
I20260621 07:55:03.911247 28248 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:03.911375 28248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:03.915791 28248 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:55:03.918268 28264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:03.919683 28248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:55:03.919853 28248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "adb63f18ce854712bea216af2d5e17fa"
format_stamp: "Formatted at 2026-06-21 07:54:42 on dist-test-slave-lbpt"
server_key: "bb2ff4e89eda4ab79c774f6657959b24"
server_key_iv: "70bc4b2dbb0373376c6e7ed9c122acdd"
server_key_version: "encryptionkey@0"
I20260621 07:55:03.920553 28248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:03.944473 28248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:03.947829 28248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:03.948058 28248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:03.956295 28248 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:33777
I20260621 07:55:03.956591 28316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:33777 every 8 connection(s)
I20260621 07:55:03.957568 28248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:55:03.961733 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 28248
I20260621 07:55:03.962169 28317 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:55:03.962209 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:36323
--local_ip_for_outbound_sockets=127.12.12.1
--tserver_master_addrs=127.12.12.62:33777
--webserver_port=41661
--webserver_interface=127.12.12.1
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:55:03.964293 28317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: Bootstrap starting.
I20260621 07:55:03.988517 28317 log.cc:824] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:03.999879 28317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:55:04.000330 28317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: Bootstrap complete.
I20260621 07:55:04.005846 28317 raft_consensus.cc:348] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } }
I20260621 07:55:04.006455 28317 raft_consensus.cc:729] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb63f18ce854712bea216af2d5e17fa, State: Initialized, Role: FOLLOWER
I20260621 07:55:04.007071 28317 consensus_queue.cc:260] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } }
I20260621 07:55:04.007220 28317 raft_consensus.cc:388] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:55:04.007298 28317 raft_consensus.cc:482] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:55:04.007401 28317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:55:04.009992 28317 raft_consensus.cc:504] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } }
I20260621 07:55:04.010381 28317 leader_election.cc:302] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adb63f18ce854712bea216af2d5e17fa; no voters:
I20260621 07:55:04.010823 28317 leader_election.cc:288] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 07:55:04.010989 28321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 2 FOLLOWER]: Leader election won for term 2
I20260621 07:55:04.011319 28321 raft_consensus.cc:686] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 2 LEADER]: Becoming Leader. State: Replica: adb63f18ce854712bea216af2d5e17fa, State: Running, Role: LEADER
I20260621 07:55:04.011658 28321 consensus_queue.cc:237] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } }
I20260621 07:55:04.012140 28317 sys_catalog.cc:564] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:55:04.013259 28323 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader adb63f18ce854712bea216af2d5e17fa. Latest consensus state: current_term: 2 leader_uuid: "adb63f18ce854712bea216af2d5e17fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } } }
I20260621 07:55:04.013259 28322 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "adb63f18ce854712bea216af2d5e17fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } } }
I20260621 07:55:04.013417 28322 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:04.013417 28323 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:04.013847 28330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:55:04.018844 28330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:55:04.019699 28330 catalog_manager.cc:1259] Loaded cluster ID: 3bcd57c673b1419ea03a4d374ed19d4e
I20260621 07:55:04.019763 28330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:55:04.022228 28330 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:55:04.022949 28330 catalog_manager.cc:5879] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: Loaded TSK: 0
I20260621 07:55:04.041529 28330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: Generated new TSK 1
I20260621 07:55:04.041818 28330 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:55:04.096096 28319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:04.096400 28319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:04.096508 28319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:04.100769 28319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:04.100867 28319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:04.100960 28319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:55:04.106539 28319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:36323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=41661
--tserver_master_addrs=127.12.12.62:33777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:04.107630 28319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:04.108765 28319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:04.120225 28345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:04.120239 28344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:04.120249 28347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:04.120981 28319 server_base.cc:1034] running on GCE node
I20260621 07:55:04.121486 28319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:04.122442 28319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:04.123653 28319 hybrid_clock.cc:648] HybridClock initialized: now 1782028504123587 us; error 73 us; skew 500 ppm
Jun 21 07:55:04 dist-test-slave-lbpt krb5kdc[27702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028504, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:04.127380 28319 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:55:04.128427 28319 webserver.cc:460] Webserver started at http://127.12.12.1:41661/ using document root <none> and password file <none>
I20260621 07:55:04.129019 28319 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:04.129137 28319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:04.133533 28319 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:55:04.136129 28354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:04.137557 28319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:55:04.137744 28319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "1e3a50e2e445470cbbfd4464e0b2aed0"
format_stamp: "Formatted at 2026-06-21 07:54:42 on dist-test-slave-lbpt"
server_key: "cd809dc1abee0c1a7f6d7d04ccfe9a0b"
server_key_iv: "d60f3881849b607b619cf8c76cfe411f"
server_key_version: "encryptionkey@0"
I20260621 07:55:04.138350 28319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:04.154043 28319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:04.157716 28319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:04.157936 28319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:04.158630 28319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:04.159996 28361 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 07:55:04.164382 28319 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 07:55:04.164464 28319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260621 07:55:04.164541 28319 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 07:55:04.166337 28319 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 07:55:04.166404 28319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260621 07:55:04.166707 28361 tablet_bootstrap.cc:492] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0: Bootstrap starting.
I20260621 07:55:04.177814 28319 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:36323
I20260621 07:55:04.177870 28468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:36323 every 8 connection(s)
I20260621 07:55:04.179230 28319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:55:04.180403 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 28319
I20260621 07:55:04.180946 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:45059
--local_ip_for_outbound_sockets=127.12.12.2
--tserver_master_addrs=127.12.12.62:33777
--webserver_port=45741
--webserver_interface=127.12.12.2
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:55:04.188892 28361 log.cc:824] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:55:04 dist-test-slave-lbpt krb5kdc[27702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028504, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:55:04.199270 28361 tablet_bootstrap.cc:492] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:55:04.199738 28361 tablet_bootstrap.cc:492] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0: Bootstrap complete.
I20260621 07:55:04.201244 28361 ts_tablet_manager.cc:1397] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.004s
I20260621 07:55:04.202564 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:04.181701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:39465 (local address 127.12.12.62:33777)
0621 07:55:04.182209 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:04.182230 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:04.182825 (+ 595us) server_negotiation.cc:407] Connection header received
0621 07:55:04.184526 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:04.184546 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:04.185086 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:04.185530 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:04.187073 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.188632 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.190673 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.191360 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.191539 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:04.196464 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:04.196470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:04.196489 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:04.196524 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:04.198532 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:04.199113 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:04.199118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:04.199122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:04.199247 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:04.200291 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:04.200294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:04.200295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:04.200663 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:04.200818 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:04.201434 (+ 616us) server_negotiation.cc:299] Negotiation successful
0621 07:55:04.201716 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":113,"threads_started":1}
I20260621 07:55:04.202564 28472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:04.181691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:39465)
0621 07:55:04.182605 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:04.182672 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:04.183999 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:04.185594 (+ 1595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:04.185603 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:04.185990 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:04.186854 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:04.186880 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.188825 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.188832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:04.190508 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:04.190525 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.191620 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.191627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:04.191878 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:04.194614 (+ 2736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:55:04.194632 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:04.196206 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:04.198664 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:04.198672 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:04.198686 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:04.198969 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:04.200016 (+ 1047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:04.200019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:04.200020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:04.200163 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:04.200789 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:04.200795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:04.201057 (+ 262us) client_negotiation.cc:769] Sending connection context
0621 07:55:04.201243 (+ 186us) client_negotiation.cc:240] Negotiation successful
0621 07:55:04.201603 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":170,"threads_started":1}
I20260621 07:55:04.204493 28361 raft_consensus.cc:348] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } }
I20260621 07:55:04.205022 28361 raft_consensus.cc:729] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e3a50e2e445470cbbfd4464e0b2aed0, State: Initialized, Role: FOLLOWER
I20260621 07:55:04.205318 28469 heartbeater.cc:347] Connected to a master server at 127.12.12.62:33777
I20260621 07:55:04.205577 28469 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:04.205549 28361 consensus_queue.cc:260] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } }
I20260621 07:55:04.206173 28469 heartbeater.cc:511] Master 127.12.12.62:33777 requested a full tablet report, sending...
I20260621 07:55:04.206651 28361 ts_tablet_manager.cc:1428] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260621 07:55:04.208786 28281 ts_manager.cc:194] Registered new tserver with Master: 1e3a50e2e445470cbbfd4464e0b2aed0 (127.12.12.1:36323)
I20260621 07:55:04.212781 28281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:39465
I20260621 07:55:04.215638 28469 heartbeater.cc:503] Master 127.12.12.62:33777 was elected leader, sending a full tablet report...
Jun 21 07:55:04 dist-test-slave-lbpt krb5kdc[27702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028503, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for kudu/127.12.12.1@KRBTEST.COM
I20260621 07:55:04.223426 28476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:04.212241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:36323 (local address 127.0.0.1:49724)
0621 07:55:04.213035 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:04.213079 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:04.213245 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:04.213770 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:04.213776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:04.214218 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:04.214327 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:04.214333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.215401 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.215405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:04.216094 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:04.216108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.216878 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.216881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:04.217013 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:04.217722 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:55:04.217739 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:04.219398 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:04.221594 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:04.221603 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:04.221608 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:04.221863 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:04.222195 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:04.222198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:04.222199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:04.222283 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:04.222859 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:04.222864 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:04.222979 (+ 115us) client_negotiation.cc:769] Sending connection context
0621 07:55:04.223107 (+ 128us) client_negotiation.cc:240] Negotiation successful
0621 07:55:04.223285 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":143,"threads_started":1}
I20260621 07:55:04.223559 28477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:04.212305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49724 (local address 127.12.12.1:36323)
0621 07:55:04.212737 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:04.212749 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:04.213204 (+ 455us) server_negotiation.cc:407] Connection header received
0621 07:55:04.213408 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:04.213413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:04.213625 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:04.213770 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:04.214473 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.215269 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.216213 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.216741 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.216853 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:04.219567 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:04.219577 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:04.219590 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:04.219619 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:04.221463 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:04.221979 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:04.221986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:04.221988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:04.222081 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:04.222373 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:04.222376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:04.222378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:04.222727 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:04.222859 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:04.223213 (+ 354us) server_negotiation.cc:299] Negotiation successful
0621 07:55:04.223395 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":112,"threads_started":1}
I20260621 07:55:04.224841 28403 tablet_service.cc:1514] Processing DeleteTablet for tablet c4e85e68992044458b6eaef0f4587380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:54:43 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:49724
I20260621 07:55:04.225914 28478 tablet_replica.cc:330] stopping tablet replica
I20260621 07:55:04.226212 28478 raft_consensus.cc:2229] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:55:04.226527 28478 raft_consensus.cc:2258] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:04.227494 28478 ts_tablet_manager.cc:1905] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:55:04.231310 28478 ts_tablet_manager.cc:1918] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:55:04.231428 28478 log.cc:1196] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal/wals/c4e85e68992044458b6eaef0f4587380
I20260621 07:55:04.231743 28478 ts_tablet_manager.cc:1939] T c4e85e68992044458b6eaef0f4587380 P 1e3a50e2e445470cbbfd4464e0b2aed0: Deleting consensus metadata
I20260621 07:55:04.233508 28265 catalog_manager.cc:4841] TS 1e3a50e2e445470cbbfd4464e0b2aed0 (127.12.12.1:36323): tablet c4e85e68992044458b6eaef0f4587380 (table test-table [id=49f2960b9f5248049c078086e2446368]) successfully deleted
W20260621 07:55:04.310978 28473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:04.311256 28473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:04.311328 28473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:04.315616 28473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:04.315676 28473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:04.315763 28473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:55:04.321089 28473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:45059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=45741
--tserver_master_addrs=127.12.12.62:33777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:04.322146 28473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:04.323217 28473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:04.333343 28483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:04.333448 28473 server_base.cc:1034] running on GCE node
W20260621 07:55:04.333344 28484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:04.333343 28486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:04.334147 28473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:04.334852 28473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:04.336045 28473 hybrid_clock.cc:648] HybridClock initialized: now 1782028504336037 us; error 48 us; skew 500 ppm
Jun 21 07:55:04 dist-test-slave-lbpt krb5kdc[27702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028504, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:04.339002 28473 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:55:04.339951 28473 webserver.cc:460] Webserver started at http://127.12.12.2:45741/ using document root <none> and password file <none>
I20260621 07:55:04.340577 28473 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:04.340698 28473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:04.344734 28473 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:55:04.347035 28493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:04.348418 28473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:55:04.348578 28473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "6f483172cbbc4357adf57e8db6356b12"
format_stamp: "Formatted at 2026-06-21 07:54:43 on dist-test-slave-lbpt"
server_key: "6895cb5b30dd8949d968643afcda516a"
server_key_iv: "456c9427d7bf4b066093395a5f94d1f3"
server_key_version: "encryptionkey@0"
I20260621 07:55:04.349223 28473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:04.372956 28473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:04.376435 28473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:04.376667 28473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:04.377333 28473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:04.378438 28500 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 07:55:04.382714 28473 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 07:55:04.382833 28473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260621 07:55:04.382936 28473 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 07:55:04.384640 28473 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 07:55:04.384709 28473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260621 07:55:04.385002 28500 tablet_bootstrap.cc:492] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: Bootstrap starting.
I20260621 07:55:04.391729 28500 log.cc:824] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:04.395388 28473 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:45059
I20260621 07:55:04.395471 28608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:45059 every 8 connection(s)
I20260621 07:55:04.396314 28473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:55:04.401183 28500 tablet_bootstrap.cc:492] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:55:04.401628 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 28473
I20260621 07:55:04.401702 28500 tablet_bootstrap.cc:492] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: Bootstrap complete.
I20260621 07:55:04.402086 28500 ts_tablet_manager.cc:1397] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20260621 07:55:04.402120 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:34081
--local_ip_for_outbound_sockets=127.12.12.3
--tserver_master_addrs=127.12.12.62:33777
--webserver_port=32983
--webserver_interface=127.12.12.3
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:55:04.405498 28500 raft_consensus.cc:348] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } }
I20260621 07:55:04.406004 28500 raft_consensus.cc:729] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f483172cbbc4357adf57e8db6356b12, State: Initialized, Role: FOLLOWER
I20260621 07:55:04.406610 28500 consensus_queue.cc:260] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } }
I20260621 07:55:04.407747 28500 ts_tablet_manager.cc:1428] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: Time spent starting tablet: real 0.006s user 0.009s sys 0.000s
Jun 21 07:55:04 dist-test-slave-lbpt krb5kdc[27702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028504, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:55:04.413758 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:04.398338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:39867 (local address 127.12.12.62:33777)
0621 07:55:04.398524 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:04.398530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:04.399231 (+ 701us) server_negotiation.cc:407] Connection header received
0621 07:55:04.400358 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:04.400360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:04.400410 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:04.400488 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:04.403082 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.403746 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.405403 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.405639 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.405732 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:04.409325 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:04.409328 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:04.409330 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:04.409359 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:04.410992 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:04.411729 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:04.411732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:04.411733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:04.411837 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:04.412318 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:04.412362 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:04.412365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:04.412583 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:04.412710 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:04.413496 (+ 786us) server_negotiation.cc:299] Negotiation successful
0621 07:55:04.413623 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 07:55:04.414983 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:04.398492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:39867)
0621 07:55:04.399013 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:04.399067 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:04.400106 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:04.400659 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:04.400669 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:04.401208 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:04.402904 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:04.402921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.403916 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.403920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:04.405255 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:04.405278 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.405765 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.405769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:04.405970 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:04.407088 (+ 1118us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:55:04.407107 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:04.409121 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:04.411120 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:04.411128 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:04.411148 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:04.411577 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:04.411940 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:04.411945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:04.411950 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:04.412156 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:04.412721 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:04.412735 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:04.413098 (+ 363us) client_negotiation.cc:769] Sending connection context
0621 07:55:04.413328 (+ 230us) client_negotiation.cc:240] Negotiation successful
0621 07:55:04.413784 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":143,"threads_started":1}
I20260621 07:55:04.416260 28609 heartbeater.cc:347] Connected to a master server at 127.12.12.62:33777
I20260621 07:55:04.416507 28609 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:04.416992 28609 heartbeater.cc:511] Master 127.12.12.62:33777 requested a full tablet report, sending...
I20260621 07:55:04.418766 28281 ts_manager.cc:194] Registered new tserver with Master: 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059)
I20260621 07:55:04.419989 28281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:39867
I20260621 07:55:04.422173 28609 heartbeater.cc:503] Master 127.12.12.62:33777 was elected leader, sending a full tablet report...
Jun 21 07:55:04 dist-test-slave-lbpt krb5kdc[27702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028503, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for kudu/127.12.12.2@KRBTEST.COM
I20260621 07:55:04.430495 28476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:04.419626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:45059 (local address 127.0.0.1:60980)
0621 07:55:04.419798 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:04.419813 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:04.419882 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:04.420788 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:04.420791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:04.420991 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:04.421089 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:04.421097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.422655 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.422659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:04.423566 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:04.423577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.424425 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.424428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:04.424508 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:04.425116 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:55:04.425134 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:04.426690 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:04.428782 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:04.428786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:04.428788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:04.429043 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:04.429363 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:04.429367 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:04.429369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:04.429461 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:04.429907 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:04.429910 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:04.430014 (+ 104us) client_negotiation.cc:769] Sending connection context
0621 07:55:04.430147 (+ 133us) client_negotiation.cc:240] Negotiation successful
0621 07:55:04.430299 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260621 07:55:04.430795 28614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:04.420027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60980 (local address 127.12.12.2:45059)
0621 07:55:04.420334 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:04.420346 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:04.420367 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:55:04.420433 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:04.420438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:04.420645 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:04.420791 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:04.421221 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.422511 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.423716 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.424309 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.424416 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:04.426832 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:04.426842 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:04.426858 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:04.426884 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:04.428663 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:04.429174 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:04.429180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:04.429181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:04.429264 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:04.429551 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:04.429553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:04.429554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:04.429805 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:04.429910 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:04.430280 (+ 370us) server_negotiation.cc:299] Negotiation successful
0621 07:55:04.430521 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":214,"thread_start_us":115,"threads_started":1}
I20260621 07:55:04.431747 28542 tablet_service.cc:1514] Processing DeleteTablet for tablet c4e85e68992044458b6eaef0f4587380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:54:43 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:60980
I20260621 07:55:04.432768 28615 tablet_replica.cc:330] stopping tablet replica
I20260621 07:55:04.433077 28615 raft_consensus.cc:2229] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:55:04.433337 28615 raft_consensus.cc:2258] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:04.434270 28615 ts_tablet_manager.cc:1905] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:55:04.437839 28615 ts_tablet_manager.cc:1918] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:55:04.437922 28615 log.cc:1196] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/c4e85e68992044458b6eaef0f4587380
I20260621 07:55:04.438199 28615 ts_tablet_manager.cc:1939] T c4e85e68992044458b6eaef0f4587380 P 6f483172cbbc4357adf57e8db6356b12: Deleting consensus metadata
I20260621 07:55:04.439796 28268 catalog_manager.cc:4841] TS 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059): tablet c4e85e68992044458b6eaef0f4587380 (table test-table [id=49f2960b9f5248049c078086e2446368]) successfully deleted
W20260621 07:55:04.525302 28612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:04.525583 28612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:04.525627 28612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:04.529768 28612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:04.529829 28612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:04.529891 28612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:55:04.535317 28612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:34081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=32983
--tserver_master_addrs=127.12.12.62:33777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:04.536381 28612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:04.537397 28612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:04.547075 28620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:04.547168 28612 server_base.cc:1034] running on GCE node
W20260621 07:55:04.547338 28621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:04.547099 28623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:04.548097 28612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:04.548769 28612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:04.549933 28612 hybrid_clock.cc:648] HybridClock initialized: now 1782028504549909 us; error 35 us; skew 500 ppm
Jun 21 07:55:04 dist-test-slave-lbpt krb5kdc[27702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028504, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:04.552943 28612 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:55:04.553968 28612 webserver.cc:460] Webserver started at http://127.12.12.3:32983/ using document root <none> and password file <none>
I20260621 07:55:04.554679 28612 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:04.554761 28612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:04.558789 28612 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:55:04.560984 28630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:04.562345 28612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:55:04.562530 28612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7"
format_stamp: "Formatted at 2026-06-21 07:54:43 on dist-test-slave-lbpt"
server_key: "63d984fc73959202d9052d2f7cd6ab66"
server_key_iv: "ee8e5af0eb7411ab15518705762a18d5"
server_key_version: "encryptionkey@0"
I20260621 07:55:04.563113 28612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:04.620324 28612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:04.624094 28612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:04.624295 28612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:04.624966 28612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:04.625809 28612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:04.625855 28612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:04.625924 28612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:04.625967 28612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:04.634980 28612 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:34081
I20260621 07:55:04.635001 28743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:34081 every 8 connection(s)
I20260621 07:55:04.636029 28612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:55:04.641623 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 28612
Jun 21 07:55:04 dist-test-slave-lbpt krb5kdc[27702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028504, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:55:04.651244 28747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:04.642389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34062 (local address 127.12.12.62:33777)
0621 07:55:04.642768 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:04.642772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:04.642789 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:55:04.642932 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:04.642934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:04.642971 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:04.643047 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:04.643926 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.644467 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.645656 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.645870 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.646013 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:04.647377 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:04.647380 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:04.647383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:04.647420 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:04.649087 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:04.649589 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:04.649592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:04.649593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:04.649805 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:04.650165 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:04.650168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:04.650170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:04.650538 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:04.650775 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:04.650952 (+ 177us) server_negotiation.cc:299] Negotiation successful
0621 07:55:04.651058 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":304,"thread_start_us":67,"threads_started":1}
I20260621 07:55:04.651618 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:04.637932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:39807 (local address 127.12.12.62:33777)
0621 07:55:04.638095 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:04.638100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:04.639189 (+ 1089us) server_negotiation.cc:407] Connection header received
0621 07:55:04.640070 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:04.640073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:04.640120 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:04.640196 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:04.641410 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.642167 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.643966 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.644235 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.644356 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:04.647230 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:04.647233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:04.647235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:04.647256 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:04.649064 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:04.649702 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:04.649705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:04.649706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:04.649826 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:04.650246 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:04.650249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:04.650251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:04.650541 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:04.650660 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:04.651345 (+ 685us) server_negotiation.cc:299] Negotiation successful
0621 07:55:04.651478 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":63}
I20260621 07:55:04.652832 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:04.638306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:39807)
0621 07:55:04.639030 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:04.639079 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:04.639884 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:04.640366 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:04.640372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:04.640734 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:04.641234 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:04.641248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.642308 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.642315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:04.643855 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:04.643869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.644392 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.644396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:04.644606 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:04.645258 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:55:04.645279 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:04.647051 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:04.649200 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:04.649206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:04.649222 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:04.649560 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:04.649939 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:04.649942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:04.649945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:04.650142 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:04.650701 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:04.650708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:04.651031 (+ 323us) client_negotiation.cc:769] Sending connection context
0621 07:55:04.651272 (+ 241us) client_negotiation.cc:240] Negotiation successful
0621 07:55:04.651882 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":114,"threads_started":1}
I20260621 07:55:04.653950 28744 heartbeater.cc:347] Connected to a master server at 127.12.12.62:33777
I20260621 07:55:04.654174 28744 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:04.654651 28744 heartbeater.cc:511] Master 127.12.12.62:33777 requested a full tablet report, sending...
I20260621 07:55:04.655572 28281 ts_manager.cc:194] Registered new tserver with Master: c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081)
I20260621 07:55:04.656090 28281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:39807
I20260621 07:55:04.665230 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:55:04.669803 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:04.665918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34068 (local address 127.12.12.62:33777)
0621 07:55:04.666061 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:04.666064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:04.666271 (+ 207us) server_negotiation.cc:407] Connection header received
0621 07:55:04.666468 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:04.666471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:04.666590 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:04.666695 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:55:04.667039 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.667579 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.668642 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.668879 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.668970 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:04.669304 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:55:04.669446 (+ 142us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0621 07:55:04.669666 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34068: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260621 07:55:04.669909 28471 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34068: token expired
W20260621 07:55:04.670316 28212 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.12.62:33777)
I20260621 07:55:04.677350 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:04.670751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.12.12.62:33777)
0621 07:55:04.670878 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:04.670880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:04.670910 (+ 30us) server_negotiation.cc:407] Connection header received
0621 07:55:04.671017 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:04.671019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:04.671057 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:04.671119 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:04.671703 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.672197 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.672961 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.673159 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.673232 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:04.674554 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:04.674557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:04.674559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:04.674583 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:04.675898 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:04.676319 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:04.676322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:04.676323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:04.676395 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:04.676646 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:04.676648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:04.676649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:04.676840 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:04.676942 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:04.677138 (+ 196us) server_negotiation.cc:299] Negotiation successful
0621 07:55:04.677230 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:55:04.679944 28281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:55:04.683224 28281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:55:04.690992 28542 tablet_service.cc:1467] Processing CreateTablet for tablet 2d1cce180e8845fbbd11a7006b64da6d (DEFAULT_TABLE table=test-table [id=eb87607d6eb64b35a6e8334b3351e8b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:04.691192 28403 tablet_service.cc:1467] Processing CreateTablet for tablet 2d1cce180e8845fbbd11a7006b64da6d (DEFAULT_TABLE table=test-table [id=eb87607d6eb64b35a6e8334b3351e8b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:04.691696 28542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d1cce180e8845fbbd11a7006b64da6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:04.691810 28403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d1cce180e8845fbbd11a7006b64da6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:04.694315 28500 tablet_bootstrap.cc:492] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12: Bootstrap starting.
I20260621 07:55:04.694475 28361 tablet_bootstrap.cc:492] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0: Bootstrap starting.
I20260621 07:55:04.695181 28361 tablet_bootstrap.cc:654] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:04.695595 28500 tablet_bootstrap.cc:654] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:04.696738 28361 tablet_bootstrap.cc:492] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0: No bootstrap required, opened a new log
I20260621 07:55:04.696753 28500 tablet_bootstrap.cc:492] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12: No bootstrap required, opened a new log
I20260621 07:55:04.696846 28361 ts_tablet_manager.cc:1397] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:55:04.696859 28500 ts_tablet_manager.cc:1397] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 07:55:04.697224 28361 raft_consensus.cc:348] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:04.697237 28500 raft_consensus.cc:348] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:04.697340 28361 raft_consensus.cc:374] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:04.697337 28500 raft_consensus.cc:374] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:04.697391 28500 raft_consensus.cc:729] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f483172cbbc4357adf57e8db6356b12, State: Initialized, Role: FOLLOWER
I20260621 07:55:04.697392 28361 raft_consensus.cc:729] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e3a50e2e445470cbbfd4464e0b2aed0, State: Initialized, Role: FOLLOWER
I20260621 07:55:04.697530 28500 consensus_queue.cc:260] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:04.697528 28361 consensus_queue.cc:260] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:04.697788 28500 ts_tablet_manager.cc:1428] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260621 07:55:04.697819 28361 ts_tablet_manager.cc:1428] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260621 07:55:04.699331 28476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:04.690112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:34081 (local address 127.0.0.1:39922)
0621 07:55:04.690219 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:04.690257 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:04.690341 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:04.691647 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:04.691650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:04.691668 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:04.691777 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:04.691784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.693162 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.693166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:04.694388 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:04.694399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.698854 (+ 4455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.698858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:04.698998 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:04.699055 (+ 57us) client_negotiation.cc:769] Sending connection context
0621 07:55:04.699136 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 07:55:04.699220 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260621 07:55:04.699432 28753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:04.690740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.12.12.3:34081)
0621 07:55:04.691106 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:04.691119 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:04.691145 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:55:04.691232 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:04.691238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:04.691469 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:04.691644 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:04.691895 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.693023 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.696171 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:04.698739 (+ 2568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:04.698873 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:04.699048 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:04.699168 (+ 120us) server_negotiation.cc:299] Negotiation successful
0621 07:55:04.699285 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":124,"threads_started":1}
I20260621 07:55:04.702473 28678 tablet_service.cc:1467] Processing CreateTablet for tablet 2d1cce180e8845fbbd11a7006b64da6d (DEFAULT_TABLE table=test-table [id=eb87607d6eb64b35a6e8334b3351e8b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:04.703536 28678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d1cce180e8845fbbd11a7006b64da6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:04.710628 28756 tablet_bootstrap.cc:492] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7: Bootstrap starting.
I20260621 07:55:04.713016 28756 tablet_bootstrap.cc:654] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:04.714247 28756 log.cc:824] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:04.716287 28756 tablet_bootstrap.cc:492] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7: No bootstrap required, opened a new log
I20260621 07:55:04.716521 28756 ts_tablet_manager.cc:1397] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:55:04.719863 28756 raft_consensus.cc:348] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:04.720145 28756 raft_consensus.cc:374] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:04.720211 28756 raft_consensus.cc:729] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b655ed1d1d4ffab5e07dd085a2dda7, State: Initialized, Role: FOLLOWER
I20260621 07:55:04.720674 28756 consensus_queue.cc:260] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:04.721386 28744 heartbeater.cc:503] Master 127.12.12.62:33777 was elected leader, sending a full tablet report...
I20260621 07:55:04.721652 28756 ts_tablet_manager.cc:1428] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260621 07:55:04.887154 28745 tablet.cc:2369] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:55:04.897589 28610 tablet.cc:2369] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:55:04.930729 28470 tablet.cc:2369] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:55:05.072110 28475 raft_consensus.cc:482] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:55:05.072418 28475 raft_consensus.cc:504] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:05.073621 28475 leader_election.cc:288] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081), 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059)
I20260621 07:55:05.077499 28753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:05.073993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:49443 (local address 127.12.12.3:34081)
0621 07:55:05.074120 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:05.074126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:05.074246 (+ 120us) server_negotiation.cc:407] Connection header received
0621 07:55:05.074438 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:05.074443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:05.074500 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:05.074582 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:05.074848 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:05.075394 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:05.076505 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:05.076907 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:05.077026 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:05.077090 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:05.077283 (+ 193us) server_negotiation.cc:299] Negotiation successful
0621 07:55:05.077358 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:55:05.077571 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:05.074156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:42171 (local address 127.12.12.2:45059)
0621 07:55:05.074479 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:05.074486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:05.074516 (+ 30us) server_negotiation.cc:407] Connection header received
0621 07:55:05.074577 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:05.074579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:05.074629 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:05.074740 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:05.075006 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:05.075553 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:05.076570 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:05.077032 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:05.077155 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:05.077341 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:05.077376 (+ 35us) server_negotiation.cc:299] Negotiation successful
0621 07:55:05.077452 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":112,"threads_started":1}
I20260621 07:55:05.077571 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:05.073984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:45059 (local address 127.12.12.1:42171)
0621 07:55:05.074307 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:05.074321 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:05.074448 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:05.074758 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:05.074761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:05.074778 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:05.074884 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:05.074889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:05.075684 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:05.075689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:05.076443 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:05.076451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:05.077159 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:05.077163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:05.077244 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:05.077263 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 07:55:05.077322 (+ 59us) client_negotiation.cc:240] Negotiation successful
0621 07:55:05.077398 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":16,"thread_start_us":64,"threads_started":1}
I20260621 07:55:05.077785 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:05.073830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:34081 (local address 127.12.12.1:49443)
0621 07:55:05.074123 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:05.074139 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:05.074271 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:05.074607 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:05.074610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:05.074626 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:05.074741 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:05.074746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:05.075512 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:05.075517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:05.076382 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:05.076391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:05.077034 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:05.077038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:05.077127 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:05.077175 (+ 48us) client_negotiation.cc:769] Sending connection context
0621 07:55:05.077268 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 07:55:05.077347 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":23,"thread_start_us":89,"threads_started":1}
I20260621 07:55:05.078226 28562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d1cce180e8845fbbd11a7006b64da6d" candidate_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f483172cbbc4357adf57e8db6356b12" is_pre_election: true
I20260621 07:55:05.078543 28698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d1cce180e8845fbbd11a7006b64da6d" candidate_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" is_pre_election: true
I20260621 07:55:05.078577 28562 raft_consensus.cc:2454] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e3a50e2e445470cbbfd4464e0b2aed0 in term 0.
I20260621 07:55:05.078820 28698 raft_consensus.cc:2454] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e3a50e2e445470cbbfd4464e0b2aed0 in term 0.
I20260621 07:55:05.079110 28358 leader_election.cc:302] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e3a50e2e445470cbbfd4464e0b2aed0, 6f483172cbbc4357adf57e8db6356b12; no voters:
I20260621 07:55:05.079437 28475 raft_consensus.cc:2781] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:55:05.079528 28475 raft_consensus.cc:482] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:55:05.079572 28475 raft_consensus.cc:3037] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:05.080593 28475 raft_consensus.cc:504] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:05.080976 28475 leader_election.cc:288] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 1 election: Requested vote from peers c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081), 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059)
I20260621 07:55:05.081465 28698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d1cce180e8845fbbd11a7006b64da6d" candidate_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7"
I20260621 07:55:05.081465 28562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d1cce180e8845fbbd11a7006b64da6d" candidate_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f483172cbbc4357adf57e8db6356b12"
I20260621 07:55:05.081597 28562 raft_consensus.cc:3037] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:05.081597 28698 raft_consensus.cc:3037] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:05.082931 28698 raft_consensus.cc:2454] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e3a50e2e445470cbbfd4464e0b2aed0 in term 1.
I20260621 07:55:05.082931 28562 raft_consensus.cc:2454] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e3a50e2e445470cbbfd4464e0b2aed0 in term 1.
I20260621 07:55:05.083263 28356 leader_election.cc:302] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e3a50e2e445470cbbfd4464e0b2aed0, c1b655ed1d1d4ffab5e07dd085a2dda7; no voters:
I20260621 07:55:05.083441 28475 raft_consensus.cc:2781] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:05.083658 28475 raft_consensus.cc:686] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 LEADER]: Becoming Leader. State: Replica: 1e3a50e2e445470cbbfd4464e0b2aed0, State: Running, Role: LEADER
I20260621 07:55:05.083940 28475 consensus_queue.cc:237] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:05.087021 28281 catalog_manager.cc:5495] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e3a50e2e445470cbbfd4464e0b2aed0 (127.12.12.1). New cstate: current_term: 1 leader_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } health_report { overall_health: HEALTHY } } }
I20260621 07:55:05.156212 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:05.152545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.12.12.1:36323)
0621 07:55:05.152882 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:05.152890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:05.152905 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:55:05.152992 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:05.152994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:05.153049 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:05.153149 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:55:05.153373 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:05.154006 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:05.154901 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:05.155120 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:05.155199 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:05.155445 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:55:05.155886 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:55:05.155973 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:05.156038 (+ 65us) server_negotiation.cc:299] Negotiation successful
0621 07:55:05.156084 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":134,"threads_started":1}
I20260621 07:55:05.160148 28698 raft_consensus.cc:1261] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 1 FOLLOWER]: Refusing update from remote peer 1e3a50e2e445470cbbfd4464e0b2aed0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:55:05.160143 28562 raft_consensus.cc:1261] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12 [term 1 FOLLOWER]: Refusing update from remote peer 1e3a50e2e445470cbbfd4464e0b2aed0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:55:05.160849 28762 consensus_queue.cc:1035] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:55:05.161027 28475 consensus_queue.cc:1035] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:55:05.165925 28766 mvcc.cc:205] Tried to move back new op lower bound from 7299188757129596928 to 7299188756826603520. Current Snapshot: MvccSnapshot[applied={T|T < 7299188757129596928}]
I20260621 07:55:05.167163 28767 mvcc.cc:205] Tried to move back new op lower bound from 7299188757129596928 to 7299188756826603520. Current Snapshot: MvccSnapshot[applied={T|T < 7299188757129596928}]
I20260621 07:55:05.174732 28281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:55:05.174923 28281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:55:05.177310 28281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: Sending DeleteTablet for 3 replicas of tablet 2d1cce180e8845fbbd11a7006b64da6d
I20260621 07:55:05.178193 28403 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d1cce180e8845fbbd11a7006b64da6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:05 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:49724
I20260621 07:55:05.178555 28542 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d1cce180e8845fbbd11a7006b64da6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:05 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:60980
I20260621 07:55:05.178592 28776 tablet_replica.cc:330] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0: stopping tablet replica
I20260621 07:55:05.178867 28776 raft_consensus.cc:2229] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:55:05.178979 28678 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d1cce180e8845fbbd11a7006b64da6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:05 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:39922
I20260621 07:55:05.179325 28776 raft_consensus.cc:2258] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:05.179522 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 28319
I20260621 07:55:05.180141 28777 tablet_replica.cc:330] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12: stopping tablet replica
I20260621 07:55:05.180263 28777 raft_consensus.cc:2229] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:55:05.180388 28777 raft_consensus.cc:2258] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:05.180752 28778 tablet_replica.cc:330] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7: stopping tablet replica
I20260621 07:55:05.181102 28778 raft_consensus.cc:2229] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:55:05.181142 28777 ts_tablet_manager.cc:1905] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:55:05.181416 28778 raft_consensus.cc:2258] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:05.182897 28778 ts_tablet_manager.cc:1905] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:55:05.183713 28777 ts_tablet_manager.cc:1918] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:55:05.183815 28777 log.cc:1196] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/2d1cce180e8845fbbd11a7006b64da6d
I20260621 07:55:05.184083 28777 ts_tablet_manager.cc:1939] T 2d1cce180e8845fbbd11a7006b64da6d P 6f483172cbbc4357adf57e8db6356b12: Deleting consensus metadata
I20260621 07:55:05.184819 28268 catalog_manager.cc:4841] TS 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059): tablet 2d1cce180e8845fbbd11a7006b64da6d (table test-table [id=eb87607d6eb64b35a6e8334b3351e8b5]) successfully deleted
I20260621 07:55:05.186190 28778 ts_tablet_manager.cc:1918] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:55:05.186295 28778 log.cc:1196] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/2d1cce180e8845fbbd11a7006b64da6d
I20260621 07:55:05.186645 28778 ts_tablet_manager.cc:1939] T 2d1cce180e8845fbbd11a7006b64da6d P c1b655ed1d1d4ffab5e07dd085a2dda7: Deleting consensus metadata
I20260621 07:55:05.188172 28266 catalog_manager.cc:4841] TS c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081): tablet 2d1cce180e8845fbbd11a7006b64da6d (table test-table [id=eb87607d6eb64b35a6e8334b3351e8b5]) successfully deleted
W20260621 07:55:05.188571 28265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.1:36323 (error 108)
I20260621 07:55:05.188865 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 28473
I20260621 07:55:05.191042 28476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:05.190737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:36323 (local address 127.0.0.1:49728)
0621 07:55:05.190908 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:05.190972 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:36323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 07:55:05.191314 28265 catalog_manager.cc:4584] TS 1e3a50e2e445470cbbfd4464e0b2aed0 (127.12.12.1:36323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d1cce180e8845fbbd11a7006b64da6d: Network error: Client connection negotiation failed: client connection to 127.12.12.1:36323: connect: Connection refused (error 111)
I20260621 07:55:05.194741 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 28612
I20260621 07:55:05.200248 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 28248
I20260621 07:55:25.207441 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:33777
--webserver_interface=127.12.12.62
--webserver_port=38481
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:33777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:25.318032 28780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:25.318359 28780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:25.318473 28780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:25.322494 28780 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 07:55:25.322577 28780 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 07:55:25.322604 28780 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 07:55:25.322628 28780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:55:25.322669 28780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:25.322715 28780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:25.322777 28780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:55:25.322817 28780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:55:25.328060 28780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:33777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:33777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=38481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:25.329208 28780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:25.330674 28780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:55:25.340134 28780 server_base.cc:1034] running on GCE node
W20260621 07:55:25.340126 28786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:25.340116 28788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:25.340106 28785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:25.340975 28780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:25.342010 28780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:25.343185 28780 hybrid_clock.cc:648] HybridClock initialized: now 1782028525343166 us; error 31 us; skew 500 ppm
Jun 21 07:55:25 dist-test-slave-lbpt krb5kdc[27702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028525, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:25.350330 28780 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:55:25.351264 28780 webserver.cc:460] Webserver started at http://127.12.12.62:38481/ using document root <none> and password file <none>
I20260621 07:55:25.351845 28780 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:25.351940 28780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:25.355752 28780 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260621 07:55:25.357810 28795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:25.359189 28780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:55:25.359329 28780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "adb63f18ce854712bea216af2d5e17fa"
format_stamp: "Formatted at 2026-06-21 07:54:42 on dist-test-slave-lbpt"
server_key: "bb2ff4e89eda4ab79c774f6657959b24"
server_key_iv: "70bc4b2dbb0373376c6e7ed9c122acdd"
server_key_version: "encryptionkey@0"
I20260621 07:55:25.359871 28780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:25.383116 28780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:25.386531 28780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:25.386782 28780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:25.394310 28780 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:33777
I20260621 07:55:25.394707 28847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:33777 every 8 connection(s)
I20260621 07:55:25.395524 28780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:55:25.399827 28848 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:55:25.401436 28848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: Bootstrap starting.
I20260621 07:55:25.404495 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 28780
I20260621 07:55:25.404991 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:36323
--local_ip_for_outbound_sockets=127.12.12.1
--tserver_master_addrs=127.12.12.62:33777
--webserver_port=41661
--webserver_interface=127.12.12.1
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:55:25.423283 28848 log.cc:824] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:25.437440 28848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:55:25.437908 28848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: Bootstrap complete.
I20260621 07:55:25.443351 28848 raft_consensus.cc:348] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } }
I20260621 07:55:25.443849 28848 raft_consensus.cc:729] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb63f18ce854712bea216af2d5e17fa, State: Initialized, Role: FOLLOWER
I20260621 07:55:25.444391 28848 consensus_queue.cc:260] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } }
I20260621 07:55:25.444535 28848 raft_consensus.cc:388] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:55:25.444591 28848 raft_consensus.cc:482] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:55:25.444693 28848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 2 FOLLOWER]: Advancing to term 3
I20260621 07:55:25.447199 28848 raft_consensus.cc:504] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } }
I20260621 07:55:25.447582 28848 leader_election.cc:302] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adb63f18ce854712bea216af2d5e17fa; no voters:
I20260621 07:55:25.447938 28848 leader_election.cc:288] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 07:55:25.448073 28852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 3 FOLLOWER]: Leader election won for term 3
I20260621 07:55:25.448385 28852 raft_consensus.cc:686] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [term 3 LEADER]: Becoming Leader. State: Replica: adb63f18ce854712bea216af2d5e17fa, State: Running, Role: LEADER
I20260621 07:55:25.448694 28852 consensus_queue.cc:237] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } }
I20260621 07:55:25.449222 28848 sys_catalog.cc:564] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:55:25.450385 28853 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "adb63f18ce854712bea216af2d5e17fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } } }
I20260621 07:55:25.450557 28853 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:25.450385 28854 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader adb63f18ce854712bea216af2d5e17fa. Latest consensus state: current_term: 3 leader_uuid: "adb63f18ce854712bea216af2d5e17fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb63f18ce854712bea216af2d5e17fa" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33777 } } }
I20260621 07:55:25.450696 28854 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:25.454771 28867 catalog_manager.cc:1259] Loaded cluster ID: 3bcd57c673b1419ea03a4d374ed19d4e
I20260621 07:55:25.454830 28867 catalog_manager.cc:1552] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: loading cluster ID for follower catalog manager: success
I20260621 07:55:25.457366 28867 catalog_manager.cc:1574] T 00000000000000000000000000000000 P adb63f18ce854712bea216af2d5e17fa: acquiring CA information for follower catalog manager: success
F20260621 07:55:25.458621 28867 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f049c6789ad google::LogMessage::Fail() at ??:0
@ 0x7f049c67b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f049c6784de google::LogMessage::Flush() at ??:0
@ 0x7f049c679a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f049f33c18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f04a6fe1989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f04a6fbe023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f04a6fbe54b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f04a6fb7652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f04a6fb69e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f04a6ffd541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f049e8bb512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f049feff6db start_thread at ??:0
@ 0x7f049b17661f clone at ??:0
W20260621 07:55:25.563966 28850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:25.564339 28850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:25.564409 28850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:25.570758 28850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:25.570852 28850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:25.570945 28850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:55:25.577646 28850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:36323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=41661
--tserver_master_addrs=127.12.12.62:33777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:25.578876 28850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:25.579932 28850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:55:25.590087 28850 server_base.cc:1034] running on GCE node
W20260621 07:55:25.590404 28888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:25.590485 28891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:25.590502 28889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:25.591171 28850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:25.592092 28850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:25.593321 28850 hybrid_clock.cc:648] HybridClock initialized: now 1782028525593294 us; error 34 us; skew 500 ppm
Jun 21 07:55:25 dist-test-slave-lbpt krb5kdc[27702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028525, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:25.596863 28850 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:55:25.597813 28850 webserver.cc:460] Webserver started at http://127.12.12.1:41661/ using document root <none> and password file <none>
I20260621 07:55:25.598404 28850 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:25.598552 28850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:25.602778 28850 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:55:25.605248 28898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:25.606850 28850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:55:25.606998 28850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "1e3a50e2e445470cbbfd4464e0b2aed0"
format_stamp: "Formatted at 2026-06-21 07:54:42 on dist-test-slave-lbpt"
server_key: "cd809dc1abee0c1a7f6d7d04ccfe9a0b"
server_key_iv: "d60f3881849b607b619cf8c76cfe411f"
server_key_version: "encryptionkey@0"
I20260621 07:55:25.607584 28850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:25.627439 28850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:25.630847 28850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:25.631057 28850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:25.631664 28850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:25.632848 28905 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 07:55:25.638350 28850 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 07:55:25.638473 28850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260621 07:55:25.638585 28850 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 07:55:25.640300 28850 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 07:55:25.640373 28850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260621 07:55:25.640681 28905 tablet_bootstrap.cc:492] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0: Bootstrap starting.
I20260621 07:55:25.651158 28850 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:36323
I20260621 07:55:25.651394 29012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:36323 every 8 connection(s)
I20260621 07:55:25.652189 28850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:55:25.653172 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 28850
I20260621 07:55:25.653676 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:45059
--local_ip_for_outbound_sockets=127.12.12.2
--tserver_master_addrs=127.12.12.62:33777
--webserver_port=45741
--webserver_interface=127.12.12.2
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:55:25.662298 28905 log.cc:824] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:25.669304 28905 tablet_bootstrap.cc:492] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:55:25.669692 28905 tablet_bootstrap.cc:492] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0: Bootstrap complete.
I20260621 07:55:25.670893 28905 ts_tablet_manager.cc:1397] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.008s
I20260621 07:55:25.673480 28905 raft_consensus.cc:348] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:25.673930 28905 raft_consensus.cc:729] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e3a50e2e445470cbbfd4464e0b2aed0, State: Initialized, Role: FOLLOWER
I20260621 07:55:25.674481 28905 consensus_queue.cc:260] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:25.675493 28905 ts_tablet_manager.cc:1428] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260621 07:55:25.700732 29015 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:55:25.654367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:48993)
0621 07:55:25.655523 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:25.655584 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:25.656640 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:25.699972 (+ 43332us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":169,"threads_started":1}
W20260621 07:55:25.701313 29013 heartbeater.cc:644] Failed to heartbeat to 127.12.12.62:33777 (0 consecutive failures): Network error: Failed to ping master at 127.12.12.62:33777: Client connection negotiation failed: client connection to 127.12.12.62:33777: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 07:55:25.702098 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:25.701847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:54883)
0621 07:55:25.701978 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:25.702040 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260621 07:55:25.702806 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:25.702620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:45181)
0621 07:55:25.702722 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:25.702763 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 07:55:25.703045 29013 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:55:25.781840 29016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:25.782104 29016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:25.782148 29016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:25.786172 29016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:25.786230 29016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:25.786332 29016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:55:25.791954 29016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:45059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=45741
--tserver_master_addrs=127.12.12.62:33777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:25.793056 29016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:25.794126 29016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:25.804232 29024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:25.804234 29023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:25.804297 29026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:25.804337 29016 server_base.cc:1034] running on GCE node
I20260621 07:55:25.805207 29016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:25.805922 29016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:25.807137 29016 hybrid_clock.cc:648] HybridClock initialized: now 1782028525807118 us; error 32 us; skew 500 ppm
Jun 21 07:55:25 dist-test-slave-lbpt krb5kdc[27702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028525, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:25.810155 29016 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:55:25.811254 29016 webserver.cc:460] Webserver started at http://127.12.12.2:45741/ using document root <none> and password file <none>
I20260621 07:55:25.811972 29016 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:25.812067 29016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:25.816061 29016 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260621 07:55:25.818306 29033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:25.819711 29016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:55:25.819860 29016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "6f483172cbbc4357adf57e8db6356b12"
format_stamp: "Formatted at 2026-06-21 07:54:43 on dist-test-slave-lbpt"
server_key: "6895cb5b30dd8949d968643afcda516a"
server_key_iv: "456c9427d7bf4b066093395a5f94d1f3"
server_key_version: "encryptionkey@0"
I20260621 07:55:25.820501 29016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:25.837771 29016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:25.841396 29016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:25.841638 29016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:25.842371 29016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:25.843289 29016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:25.843364 29016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:25.843439 29016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:25.843482 29016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:25.852989 29016 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:45059
I20260621 07:55:25.853010 29146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:45059 every 8 connection(s)
I20260621 07:55:25.854090 29016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:55:25.858000 29149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:25.856476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:58423)
0621 07:55:25.857165 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:25.857275 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":102,"threads_started":1}
W20260621 07:55:25.858505 29147 heartbeater.cc:644] Failed to heartbeat to 127.12.12.62:33777 (0 consecutive failures): Network error: Failed to ping master at 127.12.12.62:33777: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
I20260621 07:55:25.859127 29149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:25.858899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:41725)
0621 07:55:25.859013 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:25.859067 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 07:55:25.859711 29149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:25.859480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:52873)
0621 07:55:25.859582 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:25.859668 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260621 07:55:25.859884 29147 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 07:55:25.862550 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 29016
I20260621 07:55:25.862982 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:34081
--local_ip_for_outbound_sockets=127.12.12.3
--tserver_master_addrs=127.12.12.62:33777
--webserver_port=32983
--webserver_interface=127.12.12.3
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:25.977103 29150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:25.977388 29150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:25.977437 29150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:25.981427 29150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:25.981489 29150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:25.981549 29150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:55:25.986826 29150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:34081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=32983
--tserver_master_addrs=127.12.12.62:33777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:25.987885 29150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:25.988921 29150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:26.000909 29158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:26.000981 29155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:26.001381 29156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:26.002074 29150 server_base.cc:1034] running on GCE node
I20260621 07:55:26.002625 29150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:26.003341 29150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:26.004516 29150 hybrid_clock.cc:648] HybridClock initialized: now 1782028526004497 us; error 33 us; skew 500 ppm
Jun 21 07:55:26 dist-test-slave-lbpt krb5kdc[27702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028526, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:26.007647 29150 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:55:26.008646 29150 webserver.cc:460] Webserver started at http://127.12.12.3:32983/ using document root <none> and password file <none>
I20260621 07:55:26.009234 29150 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:26.009299 29150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:26.013722 29150 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:55:26.016063 29165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:26.017356 29150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:55:26.017496 29150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7"
format_stamp: "Formatted at 2026-06-21 07:54:43 on dist-test-slave-lbpt"
server_key: "63d984fc73959202d9052d2f7cd6ab66"
server_key_iv: "ee8e5af0eb7411ab15518705762a18d5"
server_key_version: "encryptionkey@0"
I20260621 07:55:26.018072 29150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:26.049960 29150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:26.053238 29150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:26.053455 29150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:26.054090 29150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:26.055019 29150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:26.055091 29150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:26.055162 29150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:26.055214 29150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:26.064531 29150 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:34081
I20260621 07:55:26.064554 29278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:34081 every 8 connection(s)
I20260621 07:55:26.065626 29150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:55:26.069450 29281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:26.067953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:41579)
0621 07:55:26.068620 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:26.068741 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":118,"threads_started":1}
I20260621 07:55:26.069880 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 29150
W20260621 07:55:26.069988 29279 heartbeater.cc:644] Failed to heartbeat to 127.12.12.62:33777 (0 consecutive failures): Network error: Failed to ping master at 127.12.12.62:33777: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
I20260621 07:55:26.070727 29281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:26.070460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:38055)
0621 07:55:26.070581 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:26.070651 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260621 07:55:26.071353 29281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:26.071158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:52655)
0621 07:55:26.071260 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:26.071303 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 07:55:26.071588 29279 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 07:55:26.704105 29283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:26.703673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:47323)
0621 07:55:26.703935 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:26.703993 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260621 07:55:26.860972 29284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:26.860515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:59503)
0621 07:55:26.860807 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:26.860870 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":69,"threads_started":1}
I20260621 07:55:27.072491 29285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:27.072089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:44825)
0621 07:55:27.072338 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:27.072392 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":83,"threads_started":1}
I20260621 07:55:27.173955 29286 raft_consensus.cc:482] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:55:27.174189 29286 raft_consensus.cc:504] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:27.175388 29286 leader_election.cc:288] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081), 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059)
Jun 21 07:55:27 dist-test-slave-lbpt krb5kdc[27702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028525, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.3@KRBTEST.COM
Jun 21 07:55:27 dist-test-slave-lbpt krb5kdc[27702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028525, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.2@KRBTEST.COM
I20260621 07:55:27.190593 29288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:27.175843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:58295 (local address 127.12.12.2:45059)
0621 07:55:27.176252 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:27.176273 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:27.176308 (+ 35us) server_negotiation.cc:407] Connection header received
0621 07:55:27.176975 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:27.176981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:27.177321 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:27.177618 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:27.179033 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:27.180240 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:27.181449 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:27.181916 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:27.182092 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:27.185123 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:27.185130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:27.185147 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:27.185187 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:27.187110 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:27.187924 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:27.187931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:27.187937 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:27.188091 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:27.188583 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:27.188587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:27.188589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:27.189127 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:27.189280 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:27.190059 (+ 779us) server_negotiation.cc:299] Negotiation successful
0621 07:55:27.190366 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":72,"threads_started":1}
I20260621 07:55:27.190596 29289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:27.175814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:55023 (local address 127.12.12.3:34081)
0621 07:55:27.176287 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:27.176311 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:27.176344 (+ 33us) server_negotiation.cc:407] Connection header received
0621 07:55:27.177016 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:27.177021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:27.177321 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:27.177631 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:27.179036 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:27.180248 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:27.181446 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:27.181914 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:27.182099 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:27.184819 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:27.184824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:27.184836 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:27.184864 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:27.187026 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:27.187924 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:27.187931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:27.187937 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:27.188090 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:27.188583 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:27.188587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:27.188589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:27.189127 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:27.189280 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:27.190060 (+ 780us) server_negotiation.cc:299] Negotiation successful
0621 07:55:27.190366 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":140,"threads_started":1}
I20260621 07:55:27.190596 29283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:27.175589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:34081 (local address 127.12.12.1:55023)
0621 07:55:27.175854 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:27.175871 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:27.175991 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:27.177688 (+ 1697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:27.177693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:27.178302 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:27.178862 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:27.178878 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:27.180393 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:27.180396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:27.181320 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:27.181330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:27.182043 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:27.182046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:27.182280 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:27.182992 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:55:27.183008 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:27.184593 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:27.187203 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:27.187211 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:27.187227 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:27.187791 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:27.188227 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:27.188231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:27.188233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:27.188477 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:27.189288 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:27.189295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:27.189695 (+ 400us) client_negotiation.cc:769] Sending connection context
0621 07:55:27.189923 (+ 228us) client_negotiation.cc:240] Negotiation successful
0621 07:55:27.190331 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":211,"spinlock_wait_cycles":30848}
I20260621 07:55:27.190596 29287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:27.175643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:45059 (local address 127.12.12.1:58295)
0621 07:55:27.175993 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:27.176007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:27.176110 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:27.177688 (+ 1578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:27.177694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:27.178302 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:27.178862 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:27.178878 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:27.180389 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:27.180392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:27.181319 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:27.181328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:27.182043 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:27.182046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:27.182285 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:27.183039 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:55:27.183051 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:27.184933 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:27.187226 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:27.187228 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:27.187230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:27.187791 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:27.188220 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:27.188223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:27.188225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:27.188481 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:27.189288 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:27.189295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:27.189695 (+ 400us) client_negotiation.cc:769] Sending connection context
0621 07:55:27.189924 (+ 229us) client_negotiation.cc:240] Negotiation successful
0621 07:55:27.190384 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":400,"spinlock_wait_cycles":88960,"thread_start_us":85,"threads_started":1}
W20260621 07:55:27.193673 28900 leader_election.cc:341] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081): Not found: Tablet not found: 2d1cce180e8845fbbd11a7006b64da6d
W20260621 07:55:27.193938 28902 leader_election.cc:341] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059): Not found: Tablet not found: 2d1cce180e8845fbbd11a7006b64da6d
I20260621 07:55:27.194036 28902 leader_election.cc:302] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e3a50e2e445470cbbfd4464e0b2aed0; no voters: 6f483172cbbc4357adf57e8db6356b12, c1b655ed1d1d4ffab5e07dd085a2dda7
I20260621 07:55:27.194344 29286 raft_consensus.cc:2726] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:55:27.705296 29290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:27.704896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:60195)
0621 07:55:27.705135 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:27.705196 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260621 07:55:27.862246 29291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:27.861815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:40507)
0621 07:55:27.862097 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:27.862157 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
I20260621 07:55:28.073645 29292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:28.073260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:36069)
0621 07:55:28.073498 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:28.073554 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260621 07:55:28.706522 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:28.706059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:41479)
0621 07:55:28.706307 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:28.706364 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260621 07:55:28.863397 29294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:28.862991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:44925)
0621 07:55:28.863248 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:28.863303 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260621 07:55:28.946167 29295 raft_consensus.cc:482] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:55:28.946347 29295 raft_consensus.cc:504] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:28.946883 29295 leader_election.cc:288] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081), 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059)
W20260621 07:55:28.947634 28900 leader_election.cc:341] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081): Not found: Tablet not found: 2d1cce180e8845fbbd11a7006b64da6d
W20260621 07:55:28.947784 28902 leader_election.cc:341] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059): Not found: Tablet not found: 2d1cce180e8845fbbd11a7006b64da6d
I20260621 07:55:28.947894 28902 leader_election.cc:302] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e3a50e2e445470cbbfd4464e0b2aed0; no voters: 6f483172cbbc4357adf57e8db6356b12, c1b655ed1d1d4ffab5e07dd085a2dda7
I20260621 07:55:28.948057 29295 raft_consensus.cc:2726] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:55:29.075117 29296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:29.074714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:55307)
0621 07:55:29.074966 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:29.075026 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
I20260621 07:55:29.707814 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:29.707379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:53713)
0621 07:55:29.707644 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:29.707704 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260621 07:55:29.864734 29298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:29.864258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:35167)
0621 07:55:29.864536 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:29.864598 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260621 07:55:30.076503 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:30.076035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:39007)
0621 07:55:30.076341 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:30.076407 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":99,"threads_started":1}
I20260621 07:55:30.709259 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:30.708782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:34719)
0621 07:55:30.709087 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:30.709155 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":86,"threads_started":1}
I20260621 07:55:30.866250 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:30.865772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:53361)
0621 07:55:30.866070 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:30.866135 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":96,"threads_started":1}
I20260621 07:55:31.077870 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:31.077419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:57559)
0621 07:55:31.077715 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:31.077771 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":96,"threads_started":1}
I20260621 07:55:31.458575 29303 raft_consensus.cc:482] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:55:31.458716 29303 raft_consensus.cc:504] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:31.459079 29303 leader_election.cc:288] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081), 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059)
W20260621 07:55:31.459918 28902 leader_election.cc:341] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059): Not found: Tablet not found: 2d1cce180e8845fbbd11a7006b64da6d
W20260621 07:55:31.460073 28900 leader_election.cc:341] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081): Not found: Tablet not found: 2d1cce180e8845fbbd11a7006b64da6d
I20260621 07:55:31.460150 28900 leader_election.cc:302] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e3a50e2e445470cbbfd4464e0b2aed0; no voters: 6f483172cbbc4357adf57e8db6356b12, c1b655ed1d1d4ffab5e07dd085a2dda7
I20260621 07:55:31.460371 29303 raft_consensus.cc:2726] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:55:31.710798 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:31.710188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:55163)
0621 07:55:31.710569 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:31.710695 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":93,"threads_started":1}
I20260621 07:55:31.867508 29305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:31.867057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:35273)
0621 07:55:31.867342 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:31.867408 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":75,"threads_started":1}
I20260621 07:55:32.079123 29306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:32.078716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:33703)
0621 07:55:32.078975 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:32.079031 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
I20260621 07:55:32.712356 29307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:32.711727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:52845)
0621 07:55:32.712190 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:32.712256 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":112,"threads_started":1}
I20260621 07:55:32.868726 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:32.868294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:37653)
0621 07:55:32.868579 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:32.868629 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":75,"threads_started":1}
I20260621 07:55:33.080547 29309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:33.080081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:60415)
0621 07:55:33.080388 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:33.080449 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":76,"threads_started":1}
I20260621 07:55:33.713655 29310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:33.713185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:46199)
0621 07:55:33.713487 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:33.713552 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":118,"threads_started":1}
I20260621 07:55:33.869844 29311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:33.869452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:46099)
0621 07:55:33.869691 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:33.869746 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260621 07:55:34.081846 29312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:34.081354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:40107)
0621 07:55:34.081646 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:34.081748 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":79,"threads_started":1}
I20260621 07:55:34.714906 29313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:34.714480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:56631)
0621 07:55:34.714724 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:34.714779 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
I20260621 07:55:34.871143 29314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:34.870694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:53127)
0621 07:55:34.870989 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:34.871048 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":90,"threads_started":1}
I20260621 07:55:35.083214 29315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:35.082773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:35441)
0621 07:55:35.083051 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:35.083115 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260621 07:55:35.506695 29316 raft_consensus.cc:482] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:55:35.506855 29316 raft_consensus.cc:504] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:35.507303 29316 leader_election.cc:288] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081), 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059)
W20260621 07:55:35.508131 28902 leader_election.cc:341] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059): Not found: Tablet not found: 2d1cce180e8845fbbd11a7006b64da6d
W20260621 07:55:35.508241 28900 leader_election.cc:341] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081): Not found: Tablet not found: 2d1cce180e8845fbbd11a7006b64da6d
I20260621 07:55:35.508358 28900 leader_election.cc:302] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e3a50e2e445470cbbfd4464e0b2aed0; no voters: 6f483172cbbc4357adf57e8db6356b12, c1b655ed1d1d4ffab5e07dd085a2dda7
I20260621 07:55:35.508554 29316 raft_consensus.cc:2726] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:55:35.716161 29317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:35.715722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:36811)
0621 07:55:35.715992 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:35.716053 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":92,"threads_started":1}
I20260621 07:55:35.872442 29318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:35.872015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:35905)
0621 07:55:35.872280 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:35.872346 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260621 07:55:36.084595 29319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:36.084097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:50411)
0621 07:55:36.084414 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:36.084497 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":110,"threads_started":1}
I20260621 07:55:36.717311 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:36.716893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:56459)
0621 07:55:36.717142 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:36.717204 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
I20260621 07:55:36.873603 29321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:36.873167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:36507)
0621 07:55:36.873435 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:36.873496 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
I20260621 07:55:37.085987 29322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:37.085499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:42389)
0621 07:55:37.085815 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:37.085884 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":76,"threads_started":1}
I20260621 07:55:37.718655 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:37.718139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:46735)
0621 07:55:37.718479 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:37.718550 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":79,"threads_started":1}
I20260621 07:55:37.875041 29324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:37.874603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:56477)
0621 07:55:37.874877 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:37.874940 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260621 07:55:38.087417 29325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:38.086913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:38801)
0621 07:55:38.087227 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:38.087275 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":108,"threads_started":1}
I20260621 07:55:38.719899 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:38.719457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:51349)
0621 07:55:38.719738 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:38.719800 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":103,"threads_started":1}
I20260621 07:55:38.759112 29327 raft_consensus.cc:482] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:55:38.759263 29327 raft_consensus.cc:504] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:38.759809 29327 leader_election.cc:288] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081), 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059)
W20260621 07:55:38.760774 28902 leader_election.cc:341] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059): Not found: Tablet not found: 2d1cce180e8845fbbd11a7006b64da6d
W20260621 07:55:38.761117 28900 leader_election.cc:341] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081): Not found: Tablet not found: 2d1cce180e8845fbbd11a7006b64da6d
I20260621 07:55:38.761243 28900 leader_election.cc:302] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e3a50e2e445470cbbfd4464e0b2aed0; no voters: 6f483172cbbc4357adf57e8db6356b12, c1b655ed1d1d4ffab5e07dd085a2dda7
I20260621 07:55:38.761432 29327 raft_consensus.cc:2726] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:55:38.876374 29328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:38.875930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:55989)
0621 07:55:38.876214 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:38.876274 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
I20260621 07:55:39.088833 29329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:39.088330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:59861)
0621 07:55:39.088652 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:39.088736 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
I20260621 07:55:39.721414 29330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:39.720878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:55487)
0621 07:55:39.721250 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:39.721308 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":110,"threads_started":1}
I20260621 07:55:39.877590 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:39.877162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:42851)
0621 07:55:39.877433 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:39.877490 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
I20260621 07:55:40.090018 29332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:40.089592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.3:41921)
0621 07:55:40.089858 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:40.089919 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260621 07:55:40.722980 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:40.722558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.1:46511)
0621 07:55:40.722824 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:40.722880 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":108,"threads_started":1}
I20260621 07:55:40.879132 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:40.878643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33777 (local address 127.12.12.2:43751)
0621 07:55:40.878971 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:40.879031 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.62:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":111,"threads_started":1}
I20260621 07:55:40.881923 29335 raft_consensus.cc:482] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:55:40.882069 29335 raft_consensus.cc:504] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b655ed1d1d4ffab5e07dd085a2dda7" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 34081 } } peers { permanent_uuid: "6f483172cbbc4357adf57e8db6356b12" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45059 } } peers { permanent_uuid: "1e3a50e2e445470cbbfd4464e0b2aed0" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 36323 } }
I20260621 07:55:40.882596 29335 leader_election.cc:288] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081), 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059)
W20260621 07:55:40.883440 28902 leader_election.cc:341] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f483172cbbc4357adf57e8db6356b12 (127.12.12.2:45059): Not found: Tablet not found: 2d1cce180e8845fbbd11a7006b64da6d
W20260621 07:55:40.883579 28900 leader_election.cc:341] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1b655ed1d1d4ffab5e07dd085a2dda7 (127.12.12.3:34081): Not found: Tablet not found: 2d1cce180e8845fbbd11a7006b64da6d
I20260621 07:55:40.883679 28900 leader_election.cc:302] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e3a50e2e445470cbbfd4464e0b2aed0; no voters: 6f483172cbbc4357adf57e8db6356b12, c1b655ed1d1d4ffab5e07dd085a2dda7
I20260621 07:55:40.883870 29335 raft_consensus.cc:2726] T 2d1cce180e8845fbbd11a7006b64da6d P 1e3a50e2e445470cbbfd4464e0b2aed0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:55:41.074214 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 28850
I20260621 07:55:41.080602 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 29016
I20260621 07:55:41.086380 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 29150
2026-06-21T07:55:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59699 ms)
[----------] 30 tests from SecurityITest (242016 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:55:41 dist-test-slave-lbpt krb5kdc[29338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:55:41 dist-test-slave-lbpt krb5kdc[29338](info): set up 2 sockets
Jun 21 07:55:41 dist-test-slave-lbpt krb5kdc[29338](info): commencing operation
krb5kdc: starting...
W20260621 07:55:42.139081 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:55:42 dist-test-slave-lbpt krb5kdc[29338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:55:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:55:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:55:42.263408 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:41837
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:41837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:42.374563 29351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:42.374907 29351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:42.374994 29351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:42.379084 29351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:55:42.379180 29351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:42.379220 29351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:42.379273 29351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:55:42.379309 29351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:55:42.384673 29351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:41837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:41837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:42.385833 29351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:42.386941 29351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:55:42.396059 29351 server_base.cc:1034] running on GCE node
W20260621 07:55:42.396144 29356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:42.396206 29357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:42.396435 29359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:42.397168 29351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:42.398396 29351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:42.399639 29351 hybrid_clock.cc:648] HybridClock initialized: now 1782028542399604 us; error 52 us; skew 500 ppm
Jun 21 07:55:42 dist-test-slave-lbpt krb5kdc[29338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028542, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:42.404675 29351 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:55:42.405664 29351 webserver.cc:460] Webserver started at http://127.12.12.62:39319/ using document root <none> and password file <none>
I20260621 07:55:42.406275 29351 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:42.406371 29351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:42.406644 29351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:42.408766 29351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "3426a46cc8e24c25baf26e4b0aa7cc54"
format_stamp: "Formatted at 2026-06-21 07:55:42 on dist-test-slave-lbpt"
server_key: "d589e48d895bdb11fd582fec9a827763"
server_key_iv: "268ff303f01a2c7ae9dc039c0a9cd4ce"
server_key_version: "encryptionkey@0"
I20260621 07:55:42.409361 29351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "3426a46cc8e24c25baf26e4b0aa7cc54"
format_stamp: "Formatted at 2026-06-21 07:55:42 on dist-test-slave-lbpt"
server_key: "d589e48d895bdb11fd582fec9a827763"
server_key_iv: "268ff303f01a2c7ae9dc039c0a9cd4ce"
server_key_version: "encryptionkey@0"
I20260621 07:55:42.413357 29351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 07:55:42.416045 29366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:42.417372 29351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:55:42.417531 29351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "3426a46cc8e24c25baf26e4b0aa7cc54"
format_stamp: "Formatted at 2026-06-21 07:55:42 on dist-test-slave-lbpt"
server_key: "d589e48d895bdb11fd582fec9a827763"
server_key_iv: "268ff303f01a2c7ae9dc039c0a9cd4ce"
server_key_version: "encryptionkey@0"
I20260621 07:55:42.417654 29351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:42.433669 29351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:42.436839 29351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:42.437075 29351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:42.444209 29351 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:41837
I20260621 07:55:42.444234 29418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:41837 every 8 connection(s)
I20260621 07:55:42.445225 29351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:55:42.448086 29419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:42.450006 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 29351
I20260621 07:55:42.450139 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:55:42.450611 12336 external_mini_cluster.cc:1428] Setting key ffa3cea7a371f13bd77205c6b0a85d49
I20260621 07:55:42.455049 29419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54: Bootstrap starting.
I20260621 07:55:42.457285 29419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:42.458083 29419 log.cc:824] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:55:42 dist-test-slave-lbpt krb5kdc[29338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:55:42.460253 29419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54: No bootstrap required, opened a new log
I20260621 07:55:42.464061 29419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3426a46cc8e24c25baf26e4b0aa7cc54" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41837 } }
I20260621 07:55:42.464356 29419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:42.464437 29419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3426a46cc8e24c25baf26e4b0aa7cc54, State: Initialized, Role: FOLLOWER
I20260621 07:55:42.464874 29419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3426a46cc8e24c25baf26e4b0aa7cc54" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41837 } }
I20260621 07:55:42.464993 29419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:55:42.465080 29419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:55:42.465171 29419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:42.465428 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:42.452236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47214 (local address 127.12.12.62:41837)
0621 07:55:42.452674 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:42.452689 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:42.452724 (+ 35us) server_negotiation.cc:407] Connection header received
0621 07:55:42.453378 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:42.453396 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:42.453723 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:42.454076 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:42.454925 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:42.455971 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:42.456899 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:42.457426 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:42.457588 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:42.460494 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:42.460498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:42.460511 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:42.460538 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:42.462183 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:42.462710 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:42.462716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:42.462721 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:42.462835 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:42.463168 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:42.463170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:42.463171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:42.463574 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:42.463792 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:42.464092 (+ 300us) server_negotiation.cc:299] Negotiation successful
0621 07:55:42.464462 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":141,"threads_started":1}
I20260621 07:55:42.466174 29419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3426a46cc8e24c25baf26e4b0aa7cc54" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41837 } }
I20260621 07:55:42.466602 29419 leader_election.cc:302] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3426a46cc8e24c25baf26e4b0aa7cc54; no voters:
I20260621 07:55:42.466938 29419 leader_election.cc:288] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:55:42.467059 29424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:42.467334 29424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [term 1 LEADER]: Becoming Leader. State: Replica: 3426a46cc8e24c25baf26e4b0aa7cc54, State: Running, Role: LEADER
I20260621 07:55:42.467712 29424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3426a46cc8e24c25baf26e4b0aa7cc54" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41837 } }
I20260621 07:55:42.468238 29419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:55:42.469419 29425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3426a46cc8e24c25baf26e4b0aa7cc54. Latest consensus state: current_term: 1 leader_uuid: "3426a46cc8e24c25baf26e4b0aa7cc54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3426a46cc8e24c25baf26e4b0aa7cc54" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41837 } } }
I20260621 07:55:42.469568 29425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:42.469931 29426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3426a46cc8e24c25baf26e4b0aa7cc54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3426a46cc8e24c25baf26e4b0aa7cc54" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41837 } } }
I20260621 07:55:42.470018 29433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:55:42.470048 29426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54 [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:42.473827 29433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:55:42.480455 29433 catalog_manager.cc:1347] Generated new cluster ID: c1f7df87d7434e089ea962e7dfe2e025
I20260621 07:55:42.480546 29433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:55:42.494889 29433 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:55:42.495918 29433 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:55:42.502381 29433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54: Generated new TSK 0
I20260621 07:55:42.503185 29433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:55:42.542886 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:41837
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:42.674796 29445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:42.675091 29445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:42.675136 29445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:42.679369 29445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:42.679435 29445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:42.679525 29445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:55:42.684921 29445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:41837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:42.686029 29445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:42.687099 29445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:42.697377 29451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:42.697429 29453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:42.697394 29450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:42.697551 29445 server_base.cc:1034] running on GCE node
I20260621 07:55:42.698283 29445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:42.699215 29445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:42.700428 29445 hybrid_clock.cc:648] HybridClock initialized: now 1782028542700395 us; error 49 us; skew 500 ppm
Jun 21 07:55:42 dist-test-slave-lbpt krb5kdc[29338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028542, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:42.704770 29445 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:55:42.706034 29445 webserver.cc:460] Webserver started at http://127.12.12.1:44397/ using document root <none> and password file <none>
I20260621 07:55:42.706879 29445 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:42.706990 29445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:42.707273 29445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:42.709753 29445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "4227656845434f6cb51e709ebe58f019"
format_stamp: "Formatted at 2026-06-21 07:55:42 on dist-test-slave-lbpt"
server_key: "d115ce5072f954998af03757547bfd54"
server_key_iv: "5ee81b477c86bd45e5b866f9f55a3b0e"
server_key_version: "encryptionkey@0"
I20260621 07:55:42.710304 29445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "4227656845434f6cb51e709ebe58f019"
format_stamp: "Formatted at 2026-06-21 07:55:42 on dist-test-slave-lbpt"
server_key: "d115ce5072f954998af03757547bfd54"
server_key_iv: "5ee81b477c86bd45e5b866f9f55a3b0e"
server_key_version: "encryptionkey@0"
I20260621 07:55:42.714226 29445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 07:55:42.716912 29460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:42.718199 29445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:55:42.718351 29445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "4227656845434f6cb51e709ebe58f019"
format_stamp: "Formatted at 2026-06-21 07:55:42 on dist-test-slave-lbpt"
server_key: "d115ce5072f954998af03757547bfd54"
server_key_iv: "5ee81b477c86bd45e5b866f9f55a3b0e"
server_key_version: "encryptionkey@0"
I20260621 07:55:42.718534 29445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:42.739805 29445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:42.743052 29445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:42.743274 29445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:42.743916 29445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:42.744848 29445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:42.744926 29445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:42.745004 29445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:42.745048 29445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:42.754462 29445 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:35457
I20260621 07:55:42.754482 29573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:35457 every 8 connection(s)
I20260621 07:55:42.755402 29445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:55:42.760813 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 29445
I20260621 07:55:42.760922 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:55:42.761231 12336 external_mini_cluster.cc:1428] Setting key fb3fe47a58d37eb3a0da1d7d7e51d77e
Jun 21 07:55:42 dist-test-slave-lbpt krb5kdc[29338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028542, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:55:42.771325 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:42.757307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:48249 (local address 127.12.12.62:41837)
0621 07:55:42.757459 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:42.757463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:42.758274 (+ 811us) server_negotiation.cc:407] Connection header received
0621 07:55:42.759712 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:42.759715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:42.759792 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:42.759886 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:42.761245 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:42.761912 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:42.763455 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:42.763890 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:42.764017 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:42.766730 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:42.766733 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:42.766735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:42.766763 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:42.768645 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:42.769362 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:42.769368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:42.769373 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:42.769478 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:42.769858 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:42.769862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:42.769863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:42.770128 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:42.770256 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:42.771074 (+ 818us) server_negotiation.cc:299] Negotiation successful
0621 07:55:42.771192 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
I20260621 07:55:42.772650 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:42.757623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:41837 (local address 127.12.12.1:48249)
0621 07:55:42.758099 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:42.758145 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:42.759465 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:42.760095 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:42.760102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:42.760497 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:42.761079 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:42.761095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:42.762063 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:42.762066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:42.763312 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:42.763322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:42.764049 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:42.764052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:42.764207 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:42.764894 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:55:42.764911 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:42.766559 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:42.768792 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:42.768798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:42.768812 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:42.769244 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:42.769572 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:42.769575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:42.769576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:42.769716 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:42.770319 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:42.770328 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:42.770761 (+ 433us) client_negotiation.cc:769] Sending connection context
0621 07:55:42.771005 (+ 244us) client_negotiation.cc:240] Negotiation successful
0621 07:55:42.771392 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":123,"threads_started":1}
I20260621 07:55:42.774025 29574 heartbeater.cc:347] Connected to a master server at 127.12.12.62:41837
I20260621 07:55:42.774461 29574 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
I20260621 07:55:42.775144 29574 heartbeater.cc:511] Master 127.12.12.62:41837 requested a full tablet report, sending...
I20260621 07:55:42.776889 29383 ts_manager.cc:194] Registered new tserver with Master: 4227656845434f6cb51e709ebe58f019 (127.12.12.1:35457)
I20260621 07:55:42.778045 29383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:48249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:55:42.794695 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:41837
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:42.909952 29579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:42.910245 29579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:42.910326 29579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:42.914513 29579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:42.914588 29579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:42.914692 29579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:55:42.920137 29579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:41837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:42.921332 29579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:42.922521 29579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:42.932379 29587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:42.932412 29584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:42.932413 29585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:42.932673 29579 server_base.cc:1034] running on GCE node
I20260621 07:55:42.933116 29579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:42.933877 29579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:42.935086 29579 hybrid_clock.cc:648] HybridClock initialized: now 1782028542935049 us; error 49 us; skew 500 ppm
Jun 21 07:55:42 dist-test-slave-lbpt krb5kdc[29338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028542, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:42.938021 29579 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:55:42.939034 29579 webserver.cc:460] Webserver started at http://127.12.12.2:39853/ using document root <none> and password file <none>
I20260621 07:55:42.939632 29579 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:42.939731 29579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:42.939981 29579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:42.942204 29579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "2ac8a26c77c948c48c345723696fbe53"
format_stamp: "Formatted at 2026-06-21 07:55:42 on dist-test-slave-lbpt"
server_key: "5db811c63935462018bac8eed13b7c23"
server_key_iv: "8ca940356453bd0d00341629305eee87"
server_key_version: "encryptionkey@0"
I20260621 07:55:42.942867 29579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "2ac8a26c77c948c48c345723696fbe53"
format_stamp: "Formatted at 2026-06-21 07:55:42 on dist-test-slave-lbpt"
server_key: "5db811c63935462018bac8eed13b7c23"
server_key_iv: "8ca940356453bd0d00341629305eee87"
server_key_version: "encryptionkey@0"
I20260621 07:55:42.946749 29579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:55:42.949290 29594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:42.950807 29579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:55:42.950958 29579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "2ac8a26c77c948c48c345723696fbe53"
format_stamp: "Formatted at 2026-06-21 07:55:42 on dist-test-slave-lbpt"
server_key: "5db811c63935462018bac8eed13b7c23"
server_key_iv: "8ca940356453bd0d00341629305eee87"
server_key_version: "encryptionkey@0"
I20260621 07:55:42.951083 29579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:42.975601 29579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:42.979132 29579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:42.979357 29579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:42.980008 29579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:42.980912 29579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:42.980989 29579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:42.981065 29579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:42.981118 29579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:42.990285 29579 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:46425
I20260621 07:55:42.990314 29707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:46425 every 8 connection(s)
I20260621 07:55:42.991294 29579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:55:42.991987 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 29579
I20260621 07:55:42.992108 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:55:42.992444 12336 external_mini_cluster.cc:1428] Setting key 77923bec131f6c0a3290e2c4fb115609
Jun 21 07:55:42 dist-test-slave-lbpt krb5kdc[29338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028542, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
I20260621 07:55:43.007800 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:42.993380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:37735 (local address 127.12.12.62:41837)
0621 07:55:42.993515 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:42.993520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:42.994977 (+ 1457us) server_negotiation.cc:407] Connection header received
0621 07:55:42.996378 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:42.996382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:42.996437 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:42.996517 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:42.997770 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:42.998687 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:42.999938 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.000417 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.000556 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.003332 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:43.003338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:43.003344 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:43.003381 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:43.005059 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:43.005650 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:43.005656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:43.005658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:43.005781 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:43.006285 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:43.006289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:43.006292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:43.006629 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:43.006739 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:43.007402 (+ 663us) server_negotiation.cc:299] Negotiation successful
0621 07:55:43.007554 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:55:43.008590 29710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:42.993802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:41837 (local address 127.12.12.2:37735)
0621 07:55:42.994812 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:42.994874 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:42.996152 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:42.996700 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:42.996708 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:42.997138 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:42.997622 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:42.997635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:42.998849 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:42.998853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:42.999797 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:42.999808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.000599 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.000602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:43.000759 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.001431 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:55:43.001448 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:43.003139 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:43.005188 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:43.005193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:43.005207 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:43.005495 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:43.005913 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:43.005916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:43.005917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:43.006072 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:43.006767 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:43.006778 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:43.007119 (+ 341us) client_negotiation.cc:769] Sending connection context
0621 07:55:43.007308 (+ 189us) client_negotiation.cc:240] Negotiation successful
0621 07:55:43.007610 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":589,"threads_started":1}
I20260621 07:55:43.010013 29708 heartbeater.cc:347] Connected to a master server at 127.12.12.62:41837
I20260621 07:55:43.010288 29708 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:43.010915 29708 heartbeater.cc:511] Master 127.12.12.62:41837 requested a full tablet report, sending...
I20260621 07:55:43.012117 29383 ts_manager.cc:194] Registered new tserver with Master: 2ac8a26c77c948c48c345723696fbe53 (127.12.12.2:46425)
I20260621 07:55:43.012737 29383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:37735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:55:43.022382 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:41837
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:43.160562 29713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:43.160866 29713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:43.160957 29713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:43.165107 29713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:43.165205 29713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:43.165313 29713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:55:43.171135 29713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:41837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:43.172237 29713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:43.173281 29713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:43.185166 29721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:43.185186 29718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:43.185575 29719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:43.185751 29713 server_base.cc:1034] running on GCE node
I20260621 07:55:43.186174 29713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:43.187043 29713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:43.188242 29713 hybrid_clock.cc:648] HybridClock initialized: now 1782028543188218 us; error 41 us; skew 500 ppm
Jun 21 07:55:43 dist-test-slave-lbpt krb5kdc[29338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028543, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:43.191385 29713 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:55:43.192425 29713 webserver.cc:460] Webserver started at http://127.12.12.3:39959/ using document root <none> and password file <none>
I20260621 07:55:43.193176 29713 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:43.193279 29713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:43.193514 29713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:43.195752 29713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "92e3e52340ca4160be92ca9ada05e273"
format_stamp: "Formatted at 2026-06-21 07:55:43 on dist-test-slave-lbpt"
server_key: "bc3edaf6caef224560b8f8abf14578bc"
server_key_iv: "d2c5f6cb0a4d6ede49c8ec9c9ae6c029"
server_key_version: "encryptionkey@0"
I20260621 07:55:43.196336 29713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "92e3e52340ca4160be92ca9ada05e273"
format_stamp: "Formatted at 2026-06-21 07:55:43 on dist-test-slave-lbpt"
server_key: "bc3edaf6caef224560b8f8abf14578bc"
server_key_iv: "d2c5f6cb0a4d6ede49c8ec9c9ae6c029"
server_key_version: "encryptionkey@0"
I20260621 07:55:43.200253 29713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:55:43.202940 29728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:43.204285 29713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:55:43.204406 29713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "92e3e52340ca4160be92ca9ada05e273"
format_stamp: "Formatted at 2026-06-21 07:55:43 on dist-test-slave-lbpt"
server_key: "bc3edaf6caef224560b8f8abf14578bc"
server_key_iv: "d2c5f6cb0a4d6ede49c8ec9c9ae6c029"
server_key_version: "encryptionkey@0"
I20260621 07:55:43.204519 29713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:43.220050 29713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:43.223322 29713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:43.223516 29713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:43.224135 29713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:43.224972 29713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:43.225016 29713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:43.225085 29713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:43.225118 29713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:43.234149 29713 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:45693
I20260621 07:55:43.234169 29841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:45693 every 8 connection(s)
I20260621 07:55:43.235127 29713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:55:43.239566 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 29713
I20260621 07:55:43.239723 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:55:43.240079 12336 external_mini_cluster.cc:1428] Setting key 9614f0dce0c5086f4a92d281db6f5296
Jun 21 07:55:43 dist-test-slave-lbpt krb5kdc[29338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028543, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:55:43.248668 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.237037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:59675 (local address 127.12.12.62:41837)
0621 07:55:43.237202 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:43.237207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:43.238300 (+ 1093us) server_negotiation.cc:407] Connection header received
0621 07:55:43.239340 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:43.239344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:43.239403 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:43.239519 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:43.240855 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.241392 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.242591 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.242812 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.242933 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.245301 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:43.245305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:43.245307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:43.245334 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:43.246658 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:43.247156 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:43.247161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:43.247163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:43.247242 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:43.247579 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:43.247584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:43.247586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:43.247767 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:43.247874 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:43.248372 (+ 498us) server_negotiation.cc:299] Negotiation successful
0621 07:55:43.248502 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 07:55:43.249553 29844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.237438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:41837 (local address 127.12.12.3:59675)
0621 07:55:43.238141 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:43.238183 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:43.239096 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:43.239724 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:43.239734 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:43.240176 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:43.240694 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:43.240709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.241522 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.241525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:43.242465 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:43.242482 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.242969 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.242973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:43.243158 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.243718 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:55:43.243735 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:43.245130 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:43.246772 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:43.246778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:43.246793 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:43.247049 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:43.247341 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:43.247345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:43.247346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:43.247483 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:43.247883 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:43.247892 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:43.248138 (+ 246us) client_negotiation.cc:769] Sending connection context
0621 07:55:43.248335 (+ 197us) client_negotiation.cc:240] Negotiation successful
0621 07:55:43.248659 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":121,"threads_started":1}
I20260621 07:55:43.250774 29842 heartbeater.cc:347] Connected to a master server at 127.12.12.62:41837
I20260621 07:55:43.250988 29842 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:43.251518 29842 heartbeater.cc:511] Master 127.12.12.62:41837 requested a full tablet report, sending...
I20260621 07:55:43.252620 29383 ts_manager.cc:194] Registered new tserver with Master: 92e3e52340ca4160be92ca9ada05e273 (127.12.12.3:45693)
I20260621 07:55:43.253435 29383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:59675
I20260621 07:55:43.253933 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:55:43.263939 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.256169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.12.12.62:41837)
0621 07:55:43.256312 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:43.256318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:43.256575 (+ 257us) server_negotiation.cc:407] Connection header received
0621 07:55:43.256684 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:43.256686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:43.256723 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:43.256791 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:43.257434 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.257948 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.258877 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.259088 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.259166 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.260478 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:43.260481 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:43.260483 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:43.260504 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:43.261997 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:43.262586 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:43.262588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:43.262590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:43.262698 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:43.263039 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:43.263042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:43.263044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:43.263265 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:43.263413 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:43.263639 (+ 226us) server_negotiation.cc:299] Negotiation successful
0621 07:55:43.263762 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:55:43.267037 29382 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:55:43.270222 29382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:55:43.285369 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.279773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.12.12.1:35457)
0621 07:55:43.280674 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:43.280682 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:43.280706 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:55:43.280780 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:43.280785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:43.281006 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:43.281153 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:43.281539 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.282617 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.284146 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.284687 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.284822 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.284979 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:43.285115 (+ 136us) server_negotiation.cc:299] Negotiation successful
0621 07:55:43.285210 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":88,"threads_started":1}
I20260621 07:55:43.285360 29855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.279501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:35457 (local address 127.0.0.1:34996)
0621 07:55:43.280385 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:43.280400 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:43.280508 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:43.281285 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:43.281288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:43.281305 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:43.281408 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:43.281415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.282791 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.282794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:43.283994 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:43.284027 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.284812 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.284816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:43.284932 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.284970 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 07:55:43.285087 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 07:55:43.285210 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":5,"thread_start_us":101,"threads_started":1}
I20260621 07:55:43.285815 29857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.279938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:46425 (local address 127.0.0.1:33042)
0621 07:55:43.280710 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:43.280727 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:43.280839 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:43.281389 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:43.281392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:43.281412 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:43.281531 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:43.281541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.282707 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.282714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:43.283942 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:43.283952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.285350 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.285357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:43.285470 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.285500 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 07:55:43.285571 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 07:55:43.285653 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":24,"spinlock_wait_cycles":6272,"thread_start_us":106,"threads_started":1}
I20260621 07:55:43.286368 29858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.280111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.12.12.2:46425)
0621 07:55:43.280492 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:43.280502 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:43.280947 (+ 445us) server_negotiation.cc:407] Connection header received
0621 07:55:43.281038 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:43.281047 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:43.281244 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:43.281399 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:43.281669 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.282563 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.284069 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.285031 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.285139 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.285282 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:43.286085 (+ 803us) server_negotiation.cc:299] Negotiation successful
0621 07:55:43.286195 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":101,"threads_started":1}
I20260621 07:55:43.287164 29853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.279084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:45693 (local address 127.0.0.1:41124)
0621 07:55:43.279488 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:43.279546 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:43.279685 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:43.280792 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:43.280798 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:43.280838 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:43.281007 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:43.281014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.282508 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.282512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:43.285293 (+ 2781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:43.285306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.286757 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.286761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:43.286870 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.286895 (+ 25us) client_negotiation.cc:769] Sending connection context
0621 07:55:43.286957 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 07:55:43.287023 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":1252,"spinlock_wait_cycles":7040,"thread_start_us":118,"threads_started":1}
I20260621 07:55:43.287770 29854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.279202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.12.12.3:45693)
0621 07:55:43.280197 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:43.280213 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:43.280238 (+ 25us) server_negotiation.cc:407] Connection header received
0621 07:55:43.280336 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:43.280342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:43.280566 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:43.280768 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:43.281133 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.282352 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.285442 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.286598 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.287435 (+ 837us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.287540 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:43.287580 (+ 40us) server_negotiation.cc:299] Negotiation successful
0621 07:55:43.287647 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886,"thread_start_us":142,"threads_started":1}
I20260621 07:55:43.288815 29508 tablet_service.cc:1467] Processing CreateTablet for tablet 752a47ae69bb4e5892498bbaef808efe (DEFAULT_TABLE table=test-table [id=4d90c11cc7d14bc6bbbd510667f72591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:43.289029 29642 tablet_service.cc:1467] Processing CreateTablet for tablet 752a47ae69bb4e5892498bbaef808efe (DEFAULT_TABLE table=test-table [id=4d90c11cc7d14bc6bbbd510667f72591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:43.289976 29642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752a47ae69bb4e5892498bbaef808efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:43.289978 29508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752a47ae69bb4e5892498bbaef808efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:43.290242 29776 tablet_service.cc:1467] Processing CreateTablet for tablet 752a47ae69bb4e5892498bbaef808efe (DEFAULT_TABLE table=test-table [id=4d90c11cc7d14bc6bbbd510667f72591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:43.291365 29776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752a47ae69bb4e5892498bbaef808efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:43.297327 29859 tablet_bootstrap.cc:492] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019: Bootstrap starting.
I20260621 07:55:43.298115 29860 tablet_bootstrap.cc:492] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53: Bootstrap starting.
I20260621 07:55:43.298743 29861 tablet_bootstrap.cc:492] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273: Bootstrap starting.
I20260621 07:55:43.300151 29859 tablet_bootstrap.cc:654] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:43.300798 29860 tablet_bootstrap.cc:654] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:43.300798 29861 tablet_bootstrap.cc:654] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:43.301178 29859 log.cc:824] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:43.301685 29860 log.cc:824] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:43.301955 29861 log.cc:824] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:43.303449 29859 tablet_bootstrap.cc:492] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019: No bootstrap required, opened a new log
I20260621 07:55:43.303606 29859 ts_tablet_manager.cc:1397] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:55:43.303793 29861 tablet_bootstrap.cc:492] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273: No bootstrap required, opened a new log
I20260621 07:55:43.303793 29860 tablet_bootstrap.cc:492] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53: No bootstrap required, opened a new log
I20260621 07:55:43.303978 29861 ts_tablet_manager.cc:1397] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:55:43.303978 29860 ts_tablet_manager.cc:1397] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:55:43.306913 29859 raft_consensus.cc:348] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3e52340ca4160be92ca9ada05e273" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45693 } } peers { permanent_uuid: "2ac8a26c77c948c48c345723696fbe53" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46425 } } peers { permanent_uuid: "4227656845434f6cb51e709ebe58f019" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35457 } }
I20260621 07:55:43.307154 29859 raft_consensus.cc:374] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:43.307206 29859 raft_consensus.cc:729] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4227656845434f6cb51e709ebe58f019, State: Initialized, Role: FOLLOWER
I20260621 07:55:43.307646 29859 consensus_queue.cc:260] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3e52340ca4160be92ca9ada05e273" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45693 } } peers { permanent_uuid: "2ac8a26c77c948c48c345723696fbe53" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46425 } } peers { permanent_uuid: "4227656845434f6cb51e709ebe58f019" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35457 } }
I20260621 07:55:43.308316 29574 heartbeater.cc:503] Master 127.12.12.62:41837 was elected leader, sending a full tablet report...
I20260621 07:55:43.308431 29860 raft_consensus.cc:348] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3e52340ca4160be92ca9ada05e273" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45693 } } peers { permanent_uuid: "2ac8a26c77c948c48c345723696fbe53" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46425 } } peers { permanent_uuid: "4227656845434f6cb51e709ebe58f019" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35457 } }
I20260621 07:55:43.308668 29859 ts_tablet_manager.cc:1428] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260621 07:55:43.308706 29860 raft_consensus.cc:374] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:43.308768 29860 raft_consensus.cc:729] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac8a26c77c948c48c345723696fbe53, State: Initialized, Role: FOLLOWER
I20260621 07:55:43.309070 29861 raft_consensus.cc:348] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3e52340ca4160be92ca9ada05e273" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45693 } } peers { permanent_uuid: "2ac8a26c77c948c48c345723696fbe53" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46425 } } peers { permanent_uuid: "4227656845434f6cb51e709ebe58f019" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35457 } }
I20260621 07:55:43.309334 29861 raft_consensus.cc:374] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:43.309396 29861 raft_consensus.cc:729] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e3e52340ca4160be92ca9ada05e273, State: Initialized, Role: FOLLOWER
I20260621 07:55:43.309355 29860 consensus_queue.cc:260] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3e52340ca4160be92ca9ada05e273" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45693 } } peers { permanent_uuid: "2ac8a26c77c948c48c345723696fbe53" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46425 } } peers { permanent_uuid: "4227656845434f6cb51e709ebe58f019" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35457 } }
I20260621 07:55:43.309861 29861 consensus_queue.cc:260] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3e52340ca4160be92ca9ada05e273" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45693 } } peers { permanent_uuid: "2ac8a26c77c948c48c345723696fbe53" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46425 } } peers { permanent_uuid: "4227656845434f6cb51e709ebe58f019" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35457 } }
I20260621 07:55:43.310003 29708 heartbeater.cc:503] Master 127.12.12.62:41837 was elected leader, sending a full tablet report...
I20260621 07:55:43.310374 29860 ts_tablet_manager.cc:1428] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260621 07:55:43.310863 29861 ts_tablet_manager.cc:1428] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260621 07:55:43.311164 29842 heartbeater.cc:503] Master 127.12.12.62:41837 was elected leader, sending a full tablet report...
I20260621 07:55:43.312175 29865 raft_consensus.cc:482] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:55:43.312335 29865 raft_consensus.cc:504] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3e52340ca4160be92ca9ada05e273" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45693 } } peers { permanent_uuid: "2ac8a26c77c948c48c345723696fbe53" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46425 } } peers { permanent_uuid: "4227656845434f6cb51e709ebe58f019" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35457 } }
I20260621 07:55:43.313555 29865 leader_election.cc:288] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92e3e52340ca4160be92ca9ada05e273 (127.12.12.3:45693), 2ac8a26c77c948c48c345723696fbe53 (127.12.12.2:46425)
I20260621 07:55:43.317485 29854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.313814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:49823 (local address 127.12.12.3:45693)
0621 07:55:43.313914 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:43.313920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:43.314194 (+ 274us) server_negotiation.cc:407] Connection header received
0621 07:55:43.314259 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:43.314262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:43.314311 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:43.314377 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:43.314803 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.315295 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.316339 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.316914 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.317015 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.317077 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:43.317309 (+ 232us) server_negotiation.cc:299] Negotiation successful
0621 07:55:43.317382 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260621 07:55:43.317503 29868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.313791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:45693 (local address 127.12.12.1:49823)
0621 07:55:43.314036 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:43.314048 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:43.314168 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:43.314514 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:43.314520 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:43.314542 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:43.314692 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:43.314698 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.315408 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.315414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:43.316212 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:43.316221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.317032 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.317036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:43.317154 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.317188 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 07:55:43.317267 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 07:55:43.317346 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
I20260621 07:55:43.318041 29869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.314140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:46425 (local address 127.12.12.1:39537)
0621 07:55:43.314473 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:43.314488 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:43.314573 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:43.314819 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:43.314823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:43.314842 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:43.314974 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:43.314980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.315723 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.315729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:43.316676 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:43.316689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.317515 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.317518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:43.317709 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.317730 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:55:43.317830 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 07:55:43.317894 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":8,"thread_start_us":104,"threads_started":1}
I20260621 07:55:43.318058 29858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.314274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:39537 (local address 127.12.12.2:46425)
0621 07:55:43.314399 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:43.314404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:43.314568 (+ 164us) server_negotiation.cc:407] Connection header received
0621 07:55:43.314651 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:43.314654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:43.314702 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:43.314808 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:43.315087 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.315610 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.316811 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.317397 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.317512 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.317565 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:43.317859 (+ 294us) server_negotiation.cc:299] Negotiation successful
0621 07:55:43.317941 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:55:43.318065 29796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752a47ae69bb4e5892498bbaef808efe" candidate_uuid: "4227656845434f6cb51e709ebe58f019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e3e52340ca4160be92ca9ada05e273" is_pre_election: true
I20260621 07:55:43.318320 29796 raft_consensus.cc:2454] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4227656845434f6cb51e709ebe58f019 in term 0.
I20260621 07:55:43.318835 29462 leader_election.cc:302] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4227656845434f6cb51e709ebe58f019, 92e3e52340ca4160be92ca9ada05e273; no voters:
I20260621 07:55:43.318835 29662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752a47ae69bb4e5892498bbaef808efe" candidate_uuid: "4227656845434f6cb51e709ebe58f019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac8a26c77c948c48c345723696fbe53" is_pre_election: true
I20260621 07:55:43.319077 29662 raft_consensus.cc:2454] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4227656845434f6cb51e709ebe58f019 in term 0.
I20260621 07:55:43.319097 29865 raft_consensus.cc:2781] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:55:43.319172 29865 raft_consensus.cc:482] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:55:43.319226 29865 raft_consensus.cc:3037] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:43.320205 29865 raft_consensus.cc:504] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3e52340ca4160be92ca9ada05e273" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45693 } } peers { permanent_uuid: "2ac8a26c77c948c48c345723696fbe53" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46425 } } peers { permanent_uuid: "4227656845434f6cb51e709ebe58f019" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35457 } }
I20260621 07:55:43.320528 29865 leader_election.cc:288] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [CANDIDATE]: Term 1 election: Requested vote from peers 92e3e52340ca4160be92ca9ada05e273 (127.12.12.3:45693), 2ac8a26c77c948c48c345723696fbe53 (127.12.12.2:46425)
I20260621 07:55:43.320865 29796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752a47ae69bb4e5892498bbaef808efe" candidate_uuid: "4227656845434f6cb51e709ebe58f019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e3e52340ca4160be92ca9ada05e273"
I20260621 07:55:43.320917 29662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752a47ae69bb4e5892498bbaef808efe" candidate_uuid: "4227656845434f6cb51e709ebe58f019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac8a26c77c948c48c345723696fbe53"
I20260621 07:55:43.321033 29662 raft_consensus.cc:3037] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:43.321023 29796 raft_consensus.cc:3037] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:43.322115 29662 raft_consensus.cc:2454] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4227656845434f6cb51e709ebe58f019 in term 1.
I20260621 07:55:43.322363 29796 raft_consensus.cc:2454] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4227656845434f6cb51e709ebe58f019 in term 1.
I20260621 07:55:43.322469 29462 leader_election.cc:302] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac8a26c77c948c48c345723696fbe53, 4227656845434f6cb51e709ebe58f019; no voters:
I20260621 07:55:43.322633 29865 raft_consensus.cc:2781] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:43.322875 29865 raft_consensus.cc:686] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [term 1 LEADER]: Becoming Leader. State: Replica: 4227656845434f6cb51e709ebe58f019, State: Running, Role: LEADER
I20260621 07:55:43.323190 29865 consensus_queue.cc:237] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3e52340ca4160be92ca9ada05e273" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45693 } } peers { permanent_uuid: "2ac8a26c77c948c48c345723696fbe53" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46425 } } peers { permanent_uuid: "4227656845434f6cb51e709ebe58f019" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35457 } }
I20260621 07:55:43.326161 29383 catalog_manager.cc:5495] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4227656845434f6cb51e709ebe58f019 (127.12.12.1). New cstate: current_term: 1 leader_uuid: "4227656845434f6cb51e709ebe58f019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3e52340ca4160be92ca9ada05e273" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ac8a26c77c948c48c345723696fbe53" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4227656845434f6cb51e709ebe58f019" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35457 } health_report { overall_health: HEALTHY } } }
I20260621 07:55:43.348484 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.345039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.12.12.1:35457)
0621 07:55:43.345173 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:43.345177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:43.345191 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:55:43.345290 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:43.345292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:43.345350 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:43.345438 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:55:43.345657 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.346185 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.347135 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.347347 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.347420 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.347678 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:55:43.348100 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:55:43.348191 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:43.348259 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 07:55:43.348308 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:55:43.354941 29662 raft_consensus.cc:1261] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53 [term 1 FOLLOWER]: Refusing update from remote peer 4227656845434f6cb51e709ebe58f019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:55:43.354943 29796 raft_consensus.cc:1261] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273 [term 1 FOLLOWER]: Refusing update from remote peer 4227656845434f6cb51e709ebe58f019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:55:43.355737 29870 consensus_queue.cc:1035] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92e3e52340ca4160be92ca9ada05e273" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:55:43.356004 29865 consensus_queue.cc:1035] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac8a26c77c948c48c345723696fbe53" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:55:43.364861 29875 mvcc.cc:205] Tried to move back new op lower bound from 7299188913573244928 to 7299188913454596096. Current Snapshot: MvccSnapshot[applied={T|T < 7299188913573244928}]
I20260621 07:55:43.365453 29873 mvcc.cc:205] Tried to move back new op lower bound from 7299188913573244928 to 7299188913454596096. Current Snapshot: MvccSnapshot[applied={T|T < 7299188913573244928 or (T in {7299188913573244928})}]
I20260621 07:55:43.366436 29874 mvcc.cc:205] Tried to move back new op lower bound from 7299188913573244928 to 7299188913454596096. Current Snapshot: MvccSnapshot[applied={T|T < 7299188913573244928}]
I20260621 07:55:43.372614 29381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:55:43.372805 29381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:55:43.375371 29381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3426a46cc8e24c25baf26e4b0aa7cc54: Sending DeleteTablet for 3 replicas of tablet 752a47ae69bb4e5892498bbaef808efe
I20260621 07:55:43.376199 29642 tablet_service.cc:1514] Processing DeleteTablet for tablet 752a47ae69bb4e5892498bbaef808efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:43 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:33042
I20260621 07:55:43.376214 29508 tablet_service.cc:1514] Processing DeleteTablet for tablet 752a47ae69bb4e5892498bbaef808efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:43 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:34996
I20260621 07:55:43.376214 29776 tablet_service.cc:1514] Processing DeleteTablet for tablet 752a47ae69bb4e5892498bbaef808efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:43 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:41124
I20260621 07:55:43.376716 29884 tablet_replica.cc:330] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53: stopping tablet replica
I20260621 07:55:43.376739 29883 tablet_replica.cc:330] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273: stopping tablet replica
I20260621 07:55:43.376972 29882 tablet_replica.cc:330] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019: stopping tablet replica
I20260621 07:55:43.377050 29883 raft_consensus.cc:2229] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:55:43.377400 29882 raft_consensus.cc:2229] T 752a47ae69bb4e5892498bbaef808efe P 4227656845434f6cb51e709ebe58f019 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:55:43.377404 29883 raft_consensus.cc:2258] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:43.377559 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 29445
I20260621 07:55:43.377976 29884 raft_consensus.cc:2229] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:55:43.378378 29884 raft_consensus.cc:2258] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:43.380045 29884 ts_tablet_manager.cc:1905] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:55:43.381145 29883 ts_tablet_manager.cc:1905] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:55:43.383409 29883 ts_tablet_manager.cc:1918] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:55:43.383533 29883 log.cc:1196] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/752a47ae69bb4e5892498bbaef808efe
I20260621 07:55:43.383612 29884 ts_tablet_manager.cc:1918] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:55:43.383716 29884 log.cc:1196] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/752a47ae69bb4e5892498bbaef808efe
I20260621 07:55:43.383921 29883 ts_tablet_manager.cc:1939] T 752a47ae69bb4e5892498bbaef808efe P 92e3e52340ca4160be92ca9ada05e273: Deleting consensus metadata
I20260621 07:55:43.384057 29884 ts_tablet_manager.cc:1939] T 752a47ae69bb4e5892498bbaef808efe P 2ac8a26c77c948c48c345723696fbe53: Deleting consensus metadata
I20260621 07:55:43.385215 29368 catalog_manager.cc:4841] TS 92e3e52340ca4160be92ca9ada05e273 (127.12.12.3:45693): tablet 752a47ae69bb4e5892498bbaef808efe (table test-table [id=4d90c11cc7d14bc6bbbd510667f72591]) successfully deleted
I20260621 07:55:43.385480 29368 catalog_manager.cc:4841] TS 2ac8a26c77c948c48c345723696fbe53 (127.12.12.2:46425): tablet 752a47ae69bb4e5892498bbaef808efe (table test-table [id=4d90c11cc7d14bc6bbbd510667f72591]) successfully deleted
W20260621 07:55:43.385763 29369 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.1:35457 (error 108)
I20260621 07:55:43.385908 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 29579
I20260621 07:55:43.388027 29853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.387655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:35457 (local address 127.0.0.1:35020)
0621 07:55:43.387828 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:43.387917 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:35457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260621 07:55:43.388242 29369 catalog_manager.cc:4584] TS 4227656845434f6cb51e709ebe58f019 (127.12.12.1:35457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 752a47ae69bb4e5892498bbaef808efe: Network error: Client connection negotiation failed: client connection to 127.12.12.1:35457: connect: Connection refused (error 111)
I20260621 07:55:43.392216 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 29713
I20260621 07:55:43.397643 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 29351
2026-06-21T07:55:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2310 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-21T07:55:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:55:43Z Disabled control of system clock
I20260621 07:55:43.427731 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:43425
--webserver_interface=127.12.12.62
--webserver_port=0
--builtin_ntp_servers=127.12.12.20:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:43425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:55:43.540270 29889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:43.540580 29889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:43.540616 29889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:43.544754 29889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:55:43.544818 29889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:43.544834 29889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:43.544852 29889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:55:43.544867 29889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:55:43.549630 29889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:43425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:43425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:43.550747 29889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:43.552244 29889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:43.561040 29894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:43.561126 29889 server_base.cc:1034] running on GCE node
W20260621 07:55:43.561088 29897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:43.561040 29895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:43.561954 29889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:43.563238 29889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:43.564440 29889 hybrid_clock.cc:648] HybridClock initialized: now 1782028543564403 us; error 52 us; skew 500 ppm
I20260621 07:55:43.566395 29889 webserver.cc:460] Webserver started at http://127.12.12.62:46213/ using document root <none> and password file <none>
I20260621 07:55:43.567061 29889 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:43.567157 29889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:43.567399 29889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:43.569487 29889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "c2cd775012184ea5ad6b2ac262f7bc4b"
format_stamp: "Formatted at 2026-06-21 07:55:43 on dist-test-slave-lbpt"
server_key: "7565eede68fb91b547f53384fa5e29ba"
server_key_iv: "a5173381dcb3cab9052c666ddc9c056f"
server_key_version: "encryptionkey@0"
I20260621 07:55:43.570056 29889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "c2cd775012184ea5ad6b2ac262f7bc4b"
format_stamp: "Formatted at 2026-06-21 07:55:43 on dist-test-slave-lbpt"
server_key: "7565eede68fb91b547f53384fa5e29ba"
server_key_iv: "a5173381dcb3cab9052c666ddc9c056f"
server_key_version: "encryptionkey@0"
I20260621 07:55:43.573817 29889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260621 07:55:43.576368 29903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:43.577757 29889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:55:43.577963 29889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "c2cd775012184ea5ad6b2ac262f7bc4b"
format_stamp: "Formatted at 2026-06-21 07:55:43 on dist-test-slave-lbpt"
server_key: "7565eede68fb91b547f53384fa5e29ba"
server_key_iv: "a5173381dcb3cab9052c666ddc9c056f"
server_key_version: "encryptionkey@0"
I20260621 07:55:43.578085 29889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:43.589598 29889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:43.590286 29889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:43.590519 29889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:43.597615 29889 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:43425
I20260621 07:55:43.597625 29955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:43425 every 8 connection(s)
I20260621 07:55:43.598611 29889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:55:43.601809 29956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:43.603593 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 29889
I20260621 07:55:43.603760 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:55:43.604079 12336 external_mini_cluster.cc:1428] Setting key 5f4fc4f442d1bb9f6ddf19aed0740390
I20260621 07:55:43.608788 29956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b: Bootstrap starting.
I20260621 07:55:43.611200 29956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:43.611922 29956 log.cc:824] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:43.613596 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.605565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33334 (local address 127.12.12.62:43425)
0621 07:55:43.606255 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:43.606271 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:43.606309 (+ 38us) server_negotiation.cc:407] Connection header received
0621 07:55:43.607072 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:43.607091 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:43.607513 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:43.607887 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:43.608316 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.609335 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.610227 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.610789 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.610977 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.611861 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:43.611871 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:43.611884 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:43.612330 (+ 446us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:43.612387 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:43.612407 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:43.612514 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:43.612628 (+ 114us) server_negotiation.cc:299] Negotiation successful
0621 07:55:43.612899 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":144,"threads_started":1}
I20260621 07:55:43.613969 29956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b: No bootstrap required, opened a new log
I20260621 07:55:43.617959 29956 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cd775012184ea5ad6b2ac262f7bc4b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43425 } }
I20260621 07:55:43.618218 29956 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:43.618299 29956 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cd775012184ea5ad6b2ac262f7bc4b, State: Initialized, Role: FOLLOWER
I20260621 07:55:43.618760 29956 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cd775012184ea5ad6b2ac262f7bc4b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43425 } }
I20260621 07:55:43.618911 29956 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:55:43.618997 29956 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:55:43.619098 29956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:43.620029 29956 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cd775012184ea5ad6b2ac262f7bc4b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43425 } }
I20260621 07:55:43.620386 29956 leader_election.cc:302] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2cd775012184ea5ad6b2ac262f7bc4b; no voters:
I20260621 07:55:43.620700 29956 leader_election.cc:288] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:55:43.620817 29961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:43.621069 29961 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [term 1 LEADER]: Becoming Leader. State: Replica: c2cd775012184ea5ad6b2ac262f7bc4b, State: Running, Role: LEADER
I20260621 07:55:43.621392 29961 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cd775012184ea5ad6b2ac262f7bc4b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43425 } }
I20260621 07:55:43.622038 29956 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:55:43.623275 29963 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2cd775012184ea5ad6b2ac262f7bc4b. Latest consensus state: current_term: 1 leader_uuid: "c2cd775012184ea5ad6b2ac262f7bc4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cd775012184ea5ad6b2ac262f7bc4b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43425 } } }
I20260621 07:55:43.623426 29963 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:43.623275 29962 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2cd775012184ea5ad6b2ac262f7bc4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cd775012184ea5ad6b2ac262f7bc4b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43425 } } }
I20260621 07:55:43.623541 29962 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:43.623914 29970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:55:43.627091 29970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:55:43.633165 29970 catalog_manager.cc:1347] Generated new cluster ID: bbc075256a0e4df79cb279055ef9d715
I20260621 07:55:43.633279 29970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:55:43.657263 29970 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:55:43.658205 29970 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:55:43.664608 29970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b: Generated new TSK 0
I20260621 07:55:43.665402 29970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:55:43.675338 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43425
--builtin_ntp_servers=127.12.12.20:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:55:43.793422 29980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:43.793708 29980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:43.793749 29980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:43.797971 29980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:43.798039 29980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:43.798095 29980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:55:43.803047 29980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:43.804188 29980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:43.805641 29980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:43.815467 29986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:43.815524 29980 server_base.cc:1034] running on GCE node
W20260621 07:55:43.815449 29988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:43.815438 29985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:43.816035 29980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:43.816789 29980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:43.817976 29980 hybrid_clock.cc:648] HybridClock initialized: now 1782028543817959 us; error 36 us; skew 500 ppm
I20260621 07:55:43.819952 29980 webserver.cc:460] Webserver started at http://127.12.12.1:39859/ using document root <none> and password file <none>
I20260621 07:55:43.820546 29980 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:43.820611 29980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:43.820830 29980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:43.822976 29980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "090008bd5c6243f1993d5bb78a82eb1f"
format_stamp: "Formatted at 2026-06-21 07:55:43 on dist-test-slave-lbpt"
server_key: "8bff790525a7199f29bdbf24dc279dee"
server_key_iv: "703ed7f76034450a44b617b6a6bce511"
server_key_version: "encryptionkey@0"
I20260621 07:55:43.823536 29980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "090008bd5c6243f1993d5bb78a82eb1f"
format_stamp: "Formatted at 2026-06-21 07:55:43 on dist-test-slave-lbpt"
server_key: "8bff790525a7199f29bdbf24dc279dee"
server_key_iv: "703ed7f76034450a44b617b6a6bce511"
server_key_version: "encryptionkey@0"
I20260621 07:55:43.827329 29980 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:55:43.829777 29994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:43.831122 29980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:55:43.831264 29980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "090008bd5c6243f1993d5bb78a82eb1f"
format_stamp: "Formatted at 2026-06-21 07:55:43 on dist-test-slave-lbpt"
server_key: "8bff790525a7199f29bdbf24dc279dee"
server_key_iv: "703ed7f76034450a44b617b6a6bce511"
server_key_version: "encryptionkey@0"
I20260621 07:55:43.831383 29980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:43.864138 29980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:43.864952 29980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:43.865177 29980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:43.865839 29980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:43.866881 29980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:43.866955 29980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:43.867033 29980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:43.867086 29980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:43.876049 29980 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:45745
I20260621 07:55:43.876096 30107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:45745 every 8 connection(s)
I20260621 07:55:43.877357 29980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:55:43.878652 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 29980
I20260621 07:55:43.878789 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:55:43.879135 12336 external_mini_cluster.cc:1428] Setting key a1d5532f0f8d33b50397950ef60db7c4
I20260621 07:55:43.881413 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43425
--builtin_ntp_servers=127.12.12.20:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:55:43.888583 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.879542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:33775 (local address 127.12.12.62:43425)
0621 07:55:43.879720 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:43.879725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:43.881169 (+ 1444us) server_negotiation.cc:407] Connection header received
0621 07:55:43.883297 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:43.883302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:43.883372 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:43.883491 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:43.884407 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.885031 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.886315 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.886637 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.886765 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.888022 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:43.888031 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:43.888034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:43.888046 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:43.888066 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:43.888075 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:43.888163 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:43.888356 (+ 193us) server_negotiation.cc:299] Negotiation successful
0621 07:55:43.888428 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 07:55:43.889506 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:43.880062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43425 (local address 127.12.12.1:33775)
0621 07:55:43.880985 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:43.881043 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:43.883028 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:43.883650 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:43.883656 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:43.883744 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:43.884267 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:43.884280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.885177 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.885181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:43.886177 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:43.886188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:43.886803 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:43.886808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:43.886988 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:43.887715 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:55:43.887716 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:43.887760 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:55:43.887832 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:43.888167 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:43.888180 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:43.888186 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:55:43.888355 (+ 169us) client_negotiation.cc:240] Negotiation successful
0621 07:55:43.888652 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":141,"threads_started":1}
I20260621 07:55:43.891024 30108 heartbeater.cc:347] Connected to a master server at 127.12.12.62:43425
I20260621 07:55:43.891296 30108 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:43.891894 30108 heartbeater.cc:511] Master 127.12.12.62:43425 requested a full tablet report, sending...
I20260621 07:55:43.893517 29920 ts_manager.cc:194] Registered new tserver with Master: 090008bd5c6243f1993d5bb78a82eb1f (127.12.12.1:45745)
I20260621 07:55:43.894941 29920 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.12.1:33775
W20260621 07:55:43.998198 30111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:43.998667 30111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:43.998767 30111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:44.007212 30111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:44.007313 30111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:44.007558 30111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:55:44.016865 30111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:44.018548 30111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:44.020619 30111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:44.031257 30117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:44.031247 30116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:44.031399 30111 server_base.cc:1034] running on GCE node
W20260621 07:55:44.031582 30119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:44.031992 30111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:44.032634 30111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:44.033838 30111 hybrid_clock.cc:648] HybridClock initialized: now 1782028544033823 us; error 31 us; skew 500 ppm
I20260621 07:55:44.035888 30111 webserver.cc:460] Webserver started at http://127.12.12.2:41759/ using document root <none> and password file <none>
I20260621 07:55:44.036468 30111 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:44.036558 30111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:44.036790 30111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:44.038956 30111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "1016b6c99fdb4a068955f415458061b1"
format_stamp: "Formatted at 2026-06-21 07:55:44 on dist-test-slave-lbpt"
server_key: "e18109871f4f6d68ea22cd18217a9812"
server_key_iv: "2f4a4bc2a3fc536bcab6ad32dd919e50"
server_key_version: "encryptionkey@0"
I20260621 07:55:44.039541 30111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "1016b6c99fdb4a068955f415458061b1"
format_stamp: "Formatted at 2026-06-21 07:55:44 on dist-test-slave-lbpt"
server_key: "e18109871f4f6d68ea22cd18217a9812"
server_key_iv: "2f4a4bc2a3fc536bcab6ad32dd919e50"
server_key_version: "encryptionkey@0"
I20260621 07:55:44.043329 30111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:55:44.045895 30125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:44.047740 30111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:55:44.047845 30111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "1016b6c99fdb4a068955f415458061b1"
format_stamp: "Formatted at 2026-06-21 07:55:44 on dist-test-slave-lbpt"
server_key: "e18109871f4f6d68ea22cd18217a9812"
server_key_iv: "2f4a4bc2a3fc536bcab6ad32dd919e50"
server_key_version: "encryptionkey@0"
I20260621 07:55:44.047938 30111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:44.074112 30111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:44.074921 30111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:44.075083 30111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:44.075639 30111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:44.076660 30111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:44.076706 30111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:44.076748 30111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:44.076763 30111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:44.085950 30111 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:40345
I20260621 07:55:44.085978 30238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:40345 every 8 connection(s)
I20260621 07:55:44.086974 30111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:55:44.088936 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 30111
I20260621 07:55:44.089083 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:55:44.089378 12336 external_mini_cluster.cc:1428] Setting key cbab23ad35654742c008e7320b50b238
I20260621 07:55:44.091430 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43425
--builtin_ntp_servers=127.12.12.20:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:55:44.097643 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.088831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:56263 (local address 127.12.12.62:43425)
0621 07:55:44.088997 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:44.089002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:44.090297 (+ 1295us) server_negotiation.cc:407] Connection header received
0621 07:55:44.091780 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:44.091786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:44.091856 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:44.091952 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:44.092853 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.093396 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.095120 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.095514 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.095613 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:44.097147 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:44.097150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:44.097152 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:44.097165 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:44.097185 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:44.097193 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:44.097260 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:44.097459 (+ 199us) server_negotiation.cc:299] Negotiation successful
0621 07:55:44.097516 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 07:55:44.099018 30241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.089356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43425 (local address 127.12.12.2:56263)
0621 07:55:44.090135 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:44.090189 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:44.091518 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:44.092105 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:44.092111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:44.092197 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:44.092666 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:44.092678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.093524 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.093529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:44.094979 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:44.094994 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.095696 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.095702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:44.095958 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:44.096798 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:55:44.096802 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:44.096867 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:55:44.096962 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:44.097288 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:44.097298 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:44.097308 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 07:55:44.097461 (+ 153us) client_negotiation.cc:240] Negotiation successful
0621 07:55:44.097816 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":141,"threads_started":1}
I20260621 07:55:44.100346 30239 heartbeater.cc:347] Connected to a master server at 127.12.12.62:43425
I20260621 07:55:44.100620 30239 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:44.101187 30239 heartbeater.cc:511] Master 127.12.12.62:43425 requested a full tablet report, sending...
I20260621 07:55:44.102300 29920 ts_manager.cc:194] Registered new tserver with Master: 1016b6c99fdb4a068955f415458061b1 (127.12.12.2:40345)
I20260621 07:55:44.102937 29920 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.12.2:56263
W20260621 07:55:44.207561 30242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:44.207891 30242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:44.207965 30242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:44.212333 30242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:44.212432 30242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:44.212531 30242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:55:44.217396 30242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:44.218616 30242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:44.220062 30242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:44.229962 30250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:44.229938 30247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:44.229969 30248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:44.230263 30242 server_base.cc:1034] running on GCE node
I20260621 07:55:44.230716 30242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:44.231353 30242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:44.232510 30242 hybrid_clock.cc:648] HybridClock initialized: now 1782028544232496 us; error 28 us; skew 500 ppm
I20260621 07:55:44.234364 30242 webserver.cc:460] Webserver started at http://127.12.12.3:37349/ using document root <none> and password file <none>
I20260621 07:55:44.235073 30242 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:44.235138 30242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:44.235399 30242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:44.237639 30242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "8b40b3fb2e2641939fe10561c34704fc"
format_stamp: "Formatted at 2026-06-21 07:55:44 on dist-test-slave-lbpt"
server_key: "b6e8a8bbed54fbb71fa5ffb4bbfb498e"
server_key_iv: "f39bf7c62f9fdcab81da7574b117ce0c"
server_key_version: "encryptionkey@0"
I20260621 07:55:44.238250 30242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "8b40b3fb2e2641939fe10561c34704fc"
format_stamp: "Formatted at 2026-06-21 07:55:44 on dist-test-slave-lbpt"
server_key: "b6e8a8bbed54fbb71fa5ffb4bbfb498e"
server_key_iv: "f39bf7c62f9fdcab81da7574b117ce0c"
server_key_version: "encryptionkey@0"
I20260621 07:55:44.242027 30242 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:55:44.244676 30256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:44.245951 30242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260621 07:55:44.246075 30242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "8b40b3fb2e2641939fe10561c34704fc"
format_stamp: "Formatted at 2026-06-21 07:55:44 on dist-test-slave-lbpt"
server_key: "b6e8a8bbed54fbb71fa5ffb4bbfb498e"
server_key_iv: "f39bf7c62f9fdcab81da7574b117ce0c"
server_key_version: "encryptionkey@0"
I20260621 07:55:44.246191 30242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:44.276758 30242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:44.277554 30242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:44.277773 30242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:44.278527 30242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:44.279506 30242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:44.279562 30242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:44.279630 30242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:44.279662 30242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:44.289155 30242 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:41391
I20260621 07:55:44.289192 30369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:41391 every 8 connection(s)
I20260621 07:55:44.290097 30242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:55:44.298880 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 30242
I20260621 07:55:44.299003 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:55:44.299317 12336 external_mini_cluster.cc:1428] Setting key 9cc28291c77ed19d358fd59e91d163a4
I20260621 07:55:44.299962 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.291995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:38025 (local address 127.12.12.62:43425)
0621 07:55:44.292152 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:44.292156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:44.293268 (+ 1112us) server_negotiation.cc:407] Connection header received
0621 07:55:44.294797 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:44.294800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:44.294864 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:44.294973 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:44.295991 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.296538 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.297722 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.297967 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.298088 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:44.299387 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:44.299409 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:44.299415 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:44.299430 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:44.299454 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:44.299464 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:44.299547 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:44.299737 (+ 190us) server_negotiation.cc:299] Negotiation successful
0621 07:55:44.299795 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 07:55:44.300959 30372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.292419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43425 (local address 127.12.12.3:38025)
0621 07:55:44.293096 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:44.293138 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:44.294563 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:44.295156 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:44.295163 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:44.295256 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:44.295838 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:44.295853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.296678 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.296687 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:44.297587 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:44.297597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.298130 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.298134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:44.298310 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:44.299044 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:55:44.299050 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:44.299125 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:55:44.299203 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:44.299560 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:44.299571 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:44.299577 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:55:44.299715 (+ 138us) client_negotiation.cc:240] Negotiation successful
0621 07:55:44.300027 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":111,"threads_started":1}
I20260621 07:55:44.302124 30370 heartbeater.cc:347] Connected to a master server at 127.12.12.62:43425
I20260621 07:55:44.302357 30370 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:44.302906 30370 heartbeater.cc:511] Master 127.12.12.62:43425 requested a full tablet report, sending...
I20260621 07:55:44.303934 29920 ts_manager.cc:194] Registered new tserver with Master: 8b40b3fb2e2641939fe10561c34704fc (127.12.12.3:41391)
I20260621 07:55:44.304495 29920 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.12.3:38025
I20260621 07:55:44.313337 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:55:44.319533 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.315544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.12.12.62:43425)
0621 07:55:44.315712 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:44.315720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:44.315908 (+ 188us) server_negotiation.cc:407] Connection header received
0621 07:55:44.316065 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:44.316068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:44.316118 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:44.316219 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:44.316505 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.317028 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.317915 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.318134 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.318251 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:44.319078 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:44.319082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:44.319085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:44.319104 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:44.319126 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:44.319138 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:44.319238 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:44.319333 (+ 95us) server_negotiation.cc:299] Negotiation successful
0621 07:55:44.319387 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 07:55:44.322276 29920 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:55:44.325547 29920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:55:44.340250 30382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.334081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50746 (local address 127.12.12.1:45745)
0621 07:55:44.334927 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:44.334934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:44.334960 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:55:44.335087 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:44.335094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:44.335316 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:44.335850 (+ 534us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:44.336000 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.337228 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.338795 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.339573 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.339696 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:44.339916 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:44.339977 (+ 61us) server_negotiation.cc:299] Negotiation successful
0621 07:55:44.340076 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":94,"threads_started":1}
I20260621 07:55:44.340241 30381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.333955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:45745 (local address 127.0.0.1:50746)
0621 07:55:44.334306 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:44.334354 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:44.334495 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:44.335504 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:44.335510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:44.335537 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:44.335686 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:44.335693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.337338 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.337341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:44.338667 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:44.338676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.339710 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.339713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:44.339791 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:44.339829 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 07:55:44.339950 (+ 121us) client_negotiation.cc:240] Negotiation successful
0621 07:55:44.340076 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":96,"thread_start_us":108,"threads_started":1}
I20260621 07:55:44.340242 30383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.334278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:40345 (local address 127.0.0.1:52336)
0621 07:55:44.334808 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:44.334821 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:44.334915 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:44.335909 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:44.335912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:44.335929 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:44.336029 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:44.336038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.337436 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.337439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:44.338902 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:44.338913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.339932 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.339937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:44.340019 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:44.340037 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 07:55:44.340092 (+ 55us) client_negotiation.cc:240] Negotiation successful
0621 07:55:44.340140 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":81,"thread_start_us":100,"threads_started":1}
I20260621 07:55:44.340404 30384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.334621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52336 (local address 127.12.12.2:40345)
0621 07:55:44.335450 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:44.335465 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:44.335488 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:55:44.335562 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:44.335567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:44.335777 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:44.335935 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:44.336141 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.337262 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.339032 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.339810 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.339912 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:44.340074 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:44.340118 (+ 44us) server_negotiation.cc:299] Negotiation successful
0621 07:55:44.340204 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":122,"threads_started":1}
I20260621 07:55:44.342120 30385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.334773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:41391 (local address 127.0.0.1:41844)
0621 07:55:44.335112 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:44.335126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:44.335202 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:44.336086 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:44.336090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:44.336112 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:44.336190 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:44.336194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.337234 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.337238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:44.338902 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:44.338913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.341721 (+ 2808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.341726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:44.341824 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:44.341847 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 07:55:44.341919 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 07:55:44.341989 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":220,"thread_start_us":95,"threads_started":1}
I20260621 07:55:44.342636 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.335156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.12.12.3:41391)
0621 07:55:44.335542 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:44.335554 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:44.335575 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:55:44.335645 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:44.335650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:44.335847 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:44.335981 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:44.336306 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.337119 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.340562 (+ 3443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.341362 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.341469 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:44.341639 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:44.342355 (+ 716us) server_negotiation.cc:299] Negotiation successful
0621 07:55:44.342486 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":83,"threads_started":1}
I20260621 07:55:44.343623 30042 tablet_service.cc:1467] Processing CreateTablet for tablet 79669eea51dc4297aed3b00df489f317 (DEFAULT_TABLE table=test-table [id=4a2b6929851e4eacbf6629a05b6ddb0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:44.343619 30173 tablet_service.cc:1467] Processing CreateTablet for tablet 79669eea51dc4297aed3b00df489f317 (DEFAULT_TABLE table=test-table [id=4a2b6929851e4eacbf6629a05b6ddb0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:44.344575 30173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79669eea51dc4297aed3b00df489f317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:44.344575 30042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79669eea51dc4297aed3b00df489f317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:44.350838 30387 tablet_bootstrap.cc:492] T 79669eea51dc4297aed3b00df489f317 P 090008bd5c6243f1993d5bb78a82eb1f: Bootstrap starting.
I20260621 07:55:44.351814 30304 tablet_service.cc:1467] Processing CreateTablet for tablet 79669eea51dc4297aed3b00df489f317 (DEFAULT_TABLE table=test-table [id=4a2b6929851e4eacbf6629a05b6ddb0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:44.352428 30388 tablet_bootstrap.cc:492] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1: Bootstrap starting.
I20260621 07:55:44.352906 30304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79669eea51dc4297aed3b00df489f317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:44.353124 30387 tablet_bootstrap.cc:654] T 79669eea51dc4297aed3b00df489f317 P 090008bd5c6243f1993d5bb78a82eb1f: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:44.353925 30387 log.cc:824] T 79669eea51dc4297aed3b00df489f317 P 090008bd5c6243f1993d5bb78a82eb1f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:44.354532 30388 tablet_bootstrap.cc:654] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:44.355324 30388 log.cc:824] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:44.356058 30387 tablet_bootstrap.cc:492] T 79669eea51dc4297aed3b00df489f317 P 090008bd5c6243f1993d5bb78a82eb1f: No bootstrap required, opened a new log
I20260621 07:55:44.356258 30387 ts_tablet_manager.cc:1397] T 79669eea51dc4297aed3b00df489f317 P 090008bd5c6243f1993d5bb78a82eb1f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:55:44.359616 30391 tablet_bootstrap.cc:492] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc: Bootstrap starting.
I20260621 07:55:44.359508 30387 raft_consensus.cc:348] T 79669eea51dc4297aed3b00df489f317 P 090008bd5c6243f1993d5bb78a82eb1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090008bd5c6243f1993d5bb78a82eb1f" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45745 } } peers { permanent_uuid: "8b40b3fb2e2641939fe10561c34704fc" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 41391 } } peers { permanent_uuid: "1016b6c99fdb4a068955f415458061b1" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40345 } }
I20260621 07:55:44.359735 30387 raft_consensus.cc:374] T 79669eea51dc4297aed3b00df489f317 P 090008bd5c6243f1993d5bb78a82eb1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:44.359803 30387 raft_consensus.cc:729] T 79669eea51dc4297aed3b00df489f317 P 090008bd5c6243f1993d5bb78a82eb1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 090008bd5c6243f1993d5bb78a82eb1f, State: Initialized, Role: FOLLOWER
I20260621 07:55:44.360327 30387 consensus_queue.cc:260] T 79669eea51dc4297aed3b00df489f317 P 090008bd5c6243f1993d5bb78a82eb1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090008bd5c6243f1993d5bb78a82eb1f" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45745 } } peers { permanent_uuid: "8b40b3fb2e2641939fe10561c34704fc" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 41391 } } peers { permanent_uuid: "1016b6c99fdb4a068955f415458061b1" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40345 } }
I20260621 07:55:44.361011 30108 heartbeater.cc:503] Master 127.12.12.62:43425 was elected leader, sending a full tablet report...
I20260621 07:55:44.361374 30387 ts_tablet_manager.cc:1428] T 79669eea51dc4297aed3b00df489f317 P 090008bd5c6243f1993d5bb78a82eb1f: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260621 07:55:44.361583 30388 tablet_bootstrap.cc:492] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1: No bootstrap required, opened a new log
I20260621 07:55:44.361770 30388 ts_tablet_manager.cc:1397] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260621 07:55:44.362490 30391 tablet_bootstrap.cc:654] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:44.363374 30391 log.cc:824] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:44.365289 30391 tablet_bootstrap.cc:492] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc: No bootstrap required, opened a new log
I20260621 07:55:44.365483 30391 ts_tablet_manager.cc:1397] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260621 07:55:44.365957 30388 raft_consensus.cc:348] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090008bd5c6243f1993d5bb78a82eb1f" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45745 } } peers { permanent_uuid: "8b40b3fb2e2641939fe10561c34704fc" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 41391 } } peers { permanent_uuid: "1016b6c99fdb4a068955f415458061b1" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40345 } }
I20260621 07:55:44.366156 30388 raft_consensus.cc:374] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:44.366212 30388 raft_consensus.cc:729] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1016b6c99fdb4a068955f415458061b1, State: Initialized, Role: FOLLOWER
I20260621 07:55:44.366675 30388 consensus_queue.cc:260] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090008bd5c6243f1993d5bb78a82eb1f" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45745 } } peers { permanent_uuid: "8b40b3fb2e2641939fe10561c34704fc" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 41391 } } peers { permanent_uuid: "1016b6c99fdb4a068955f415458061b1" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40345 } }
I20260621 07:55:44.367326 30239 heartbeater.cc:503] Master 127.12.12.62:43425 was elected leader, sending a full tablet report...
I20260621 07:55:44.367566 30388 ts_tablet_manager.cc:1428] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260621 07:55:44.369287 30391 raft_consensus.cc:348] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090008bd5c6243f1993d5bb78a82eb1f" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45745 } } peers { permanent_uuid: "8b40b3fb2e2641939fe10561c34704fc" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 41391 } } peers { permanent_uuid: "1016b6c99fdb4a068955f415458061b1" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40345 } }
I20260621 07:55:44.369494 30391 raft_consensus.cc:374] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:44.369549 30391 raft_consensus.cc:729] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b40b3fb2e2641939fe10561c34704fc, State: Initialized, Role: FOLLOWER
I20260621 07:55:44.369964 30391 consensus_queue.cc:260] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090008bd5c6243f1993d5bb78a82eb1f" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45745 } } peers { permanent_uuid: "8b40b3fb2e2641939fe10561c34704fc" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 41391 } } peers { permanent_uuid: "1016b6c99fdb4a068955f415458061b1" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40345 } }
I20260621 07:55:44.370815 30370 heartbeater.cc:503] Master 127.12.12.62:43425 was elected leader, sending a full tablet report...
I20260621 07:55:44.371008 30391 ts_tablet_manager.cc:1428] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
W20260621 07:55:44.378610 30109 tablet.cc:2369] T 79669eea51dc4297aed3b00df489f317 P 090008bd5c6243f1993d5bb78a82eb1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:55:44.507273 30394 raft_consensus.cc:482] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:55:44.507555 30394 raft_consensus.cc:504] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090008bd5c6243f1993d5bb78a82eb1f" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45745 } } peers { permanent_uuid: "8b40b3fb2e2641939fe10561c34704fc" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 41391 } } peers { permanent_uuid: "1016b6c99fdb4a068955f415458061b1" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40345 } }
I20260621 07:55:44.508719 30394 leader_election.cc:288] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 090008bd5c6243f1993d5bb78a82eb1f (127.12.12.1:45745), 8b40b3fb2e2641939fe10561c34704fc (127.12.12.3:41391)
I20260621 07:55:44.513070 30382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.509280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:59181 (local address 127.12.12.1:45745)
0621 07:55:44.509427 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:44.509432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:44.509449 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:55:44.509581 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:44.509584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:44.509638 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:44.509723 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:44.510187 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.511051 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.512097 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.512508 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.512629 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:44.512689 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:44.512872 (+ 183us) server_negotiation.cc:299] Negotiation successful
0621 07:55:44.512940 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 07:55:44.513072 30241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.509120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:45745 (local address 127.12.12.2:59181)
0621 07:55:44.509290 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:44.509306 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:44.509440 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:44.509900 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:44.509906 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:44.509935 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:44.510082 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:44.510093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.511156 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.511162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:44.511970 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:44.511979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.512632 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.512636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:44.512735 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:44.512762 (+ 27us) client_negotiation.cc:769] Sending connection context
0621 07:55:44.512856 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 07:55:44.512931 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":1,"spinlock_wait_cycles":768}
I20260621 07:55:44.513362 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.509581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:58597 (local address 127.12.12.3:41391)
0621 07:55:44.509962 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:44.510016 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:44.510037 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:55:44.510129 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:44.510134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:44.510231 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:44.510356 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:44.510788 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.511383 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.512415 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.512818 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.512940 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:44.512997 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:44.513180 (+ 183us) server_negotiation.cc:299] Negotiation successful
0621 07:55:44.513247 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260621 07:55:44.513381 30396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.509438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:41391 (local address 127.12.12.2:58597)
0621 07:55:44.509866 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:44.509881 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:44.509972 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:44.510398 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:44.510401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:44.510460 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:44.510628 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:44.510648 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.511498 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.511504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:44.512280 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:44.512289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.512948 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.512951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:44.513047 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:44.513073 (+ 26us) client_negotiation.cc:769] Sending connection context
0621 07:55:44.513163 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 07:55:44.513229 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":175,"threads_started":1}
I20260621 07:55:44.513876 30062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79669eea51dc4297aed3b00df489f317" candidate_uuid: "1016b6c99fdb4a068955f415458061b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "090008bd5c6243f1993d5bb78a82eb1f" is_pre_election: true
I20260621 07:55:44.514214 30062 raft_consensus.cc:2454] T 79669eea51dc4297aed3b00df489f317 P 090008bd5c6243f1993d5bb78a82eb1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1016b6c99fdb4a068955f415458061b1 in term 0.
I20260621 07:55:44.514357 30324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79669eea51dc4297aed3b00df489f317" candidate_uuid: "1016b6c99fdb4a068955f415458061b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40b3fb2e2641939fe10561c34704fc" is_pre_election: true
I20260621 07:55:44.514665 30324 raft_consensus.cc:2454] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1016b6c99fdb4a068955f415458061b1 in term 0.
I20260621 07:55:44.514775 30128 leader_election.cc:302] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 090008bd5c6243f1993d5bb78a82eb1f, 1016b6c99fdb4a068955f415458061b1; no voters:
I20260621 07:55:44.515141 30394 raft_consensus.cc:2781] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:55:44.515242 30394 raft_consensus.cc:482] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:55:44.515298 30394 raft_consensus.cc:3037] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:44.516323 30394 raft_consensus.cc:504] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090008bd5c6243f1993d5bb78a82eb1f" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45745 } } peers { permanent_uuid: "8b40b3fb2e2641939fe10561c34704fc" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 41391 } } peers { permanent_uuid: "1016b6c99fdb4a068955f415458061b1" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40345 } }
I20260621 07:55:44.516656 30394 leader_election.cc:288] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [CANDIDATE]: Term 1 election: Requested vote from peers 090008bd5c6243f1993d5bb78a82eb1f (127.12.12.1:45745), 8b40b3fb2e2641939fe10561c34704fc (127.12.12.3:41391)
I20260621 07:55:44.517011 30062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79669eea51dc4297aed3b00df489f317" candidate_uuid: "1016b6c99fdb4a068955f415458061b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "090008bd5c6243f1993d5bb78a82eb1f"
I20260621 07:55:44.517143 30062 raft_consensus.cc:3037] T 79669eea51dc4297aed3b00df489f317 P 090008bd5c6243f1993d5bb78a82eb1f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:44.517210 30324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79669eea51dc4297aed3b00df489f317" candidate_uuid: "1016b6c99fdb4a068955f415458061b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40b3fb2e2641939fe10561c34704fc"
I20260621 07:55:44.517345 30324 raft_consensus.cc:3037] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:44.518260 30062 raft_consensus.cc:2454] T 79669eea51dc4297aed3b00df489f317 P 090008bd5c6243f1993d5bb78a82eb1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1016b6c99fdb4a068955f415458061b1 in term 1.
I20260621 07:55:44.518518 30324 raft_consensus.cc:2454] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1016b6c99fdb4a068955f415458061b1 in term 1.
I20260621 07:55:44.518555 30128 leader_election.cc:302] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 090008bd5c6243f1993d5bb78a82eb1f, 1016b6c99fdb4a068955f415458061b1; no voters:
I20260621 07:55:44.518708 30394 raft_consensus.cc:2781] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:44.518937 30394 raft_consensus.cc:686] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [term 1 LEADER]: Becoming Leader. State: Replica: 1016b6c99fdb4a068955f415458061b1, State: Running, Role: LEADER
I20260621 07:55:44.519281 30394 consensus_queue.cc:237] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090008bd5c6243f1993d5bb78a82eb1f" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45745 } } peers { permanent_uuid: "8b40b3fb2e2641939fe10561c34704fc" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 41391 } } peers { permanent_uuid: "1016b6c99fdb4a068955f415458061b1" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40345 } }
I20260621 07:55:44.522662 29920 catalog_manager.cc:5495] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1016b6c99fdb4a068955f415458061b1 (127.12.12.2). New cstate: current_term: 1 leader_uuid: "1016b6c99fdb4a068955f415458061b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090008bd5c6243f1993d5bb78a82eb1f" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40b3fb2e2641939fe10561c34704fc" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 41391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1016b6c99fdb4a068955f415458061b1" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 40345 } health_report { overall_health: HEALTHY } } }
W20260621 07:55:44.541121 30371 tablet.cc:2369] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:55:44.575332 30384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.571644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52348 (local address 127.12.12.2:40345)
0621 07:55:44.571798 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:44.571803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:44.571820 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:55:44.571917 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:44.571922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:44.571983 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:44.572105 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:55:44.572361 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.572880 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.573774 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:44.573998 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:44.574109 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:44.574468 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:55:44.574915 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:55:44.575057 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:44.575127 (+ 70us) server_negotiation.cc:299] Negotiation successful
0621 07:55:44.575187 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:55:44.581614 30062 raft_consensus.cc:1261] T 79669eea51dc4297aed3b00df489f317 P 090008bd5c6243f1993d5bb78a82eb1f [term 1 FOLLOWER]: Refusing update from remote peer 1016b6c99fdb4a068955f415458061b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:55:44.581614 30324 raft_consensus.cc:1261] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc [term 1 FOLLOWER]: Refusing update from remote peer 1016b6c99fdb4a068955f415458061b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:55:44.582513 30397 consensus_queue.cc:1035] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40b3fb2e2641939fe10561c34704fc" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 41391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:55:44.582705 30394 consensus_queue.cc:1035] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "090008bd5c6243f1993d5bb78a82eb1f" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 45745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 07:55:44.588034 30240 tablet.cc:2369] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:55:44.590723 30400 mvcc.cc:205] Tried to move back new op lower bound from 7299188918597918720 to 7299188918353723392. Current Snapshot: MvccSnapshot[applied={T|T < 7299188918597918720}]
I20260621 07:55:44.593281 30401 mvcc.cc:205] Tried to move back new op lower bound from 7299188918597918720 to 7299188918353723392. Current Snapshot: MvccSnapshot[applied={T|T < 7299188918597918720}]
I20260621 07:55:44.599653 29919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:55:44.599820 29919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:55:44.602355 29919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2cd775012184ea5ad6b2ac262f7bc4b: Sending DeleteTablet for 3 replicas of tablet 79669eea51dc4297aed3b00df489f317
I20260621 07:55:44.603117 30173 tablet_service.cc:1514] Processing DeleteTablet for tablet 79669eea51dc4297aed3b00df489f317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:44 UTC) from {username='slave'} at 127.0.0.1:52336
I20260621 07:55:44.603645 30410 tablet_replica.cc:330] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1: stopping tablet replica
I20260621 07:55:44.603624 30042 tablet_service.cc:1514] Processing DeleteTablet for tablet 79669eea51dc4297aed3b00df489f317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:44 UTC) from {username='slave'} at 127.0.0.1:50746
I20260621 07:55:44.604079 30410 raft_consensus.cc:2229] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:55:44.604215 30304 tablet_service.cc:1514] Processing DeleteTablet for tablet 79669eea51dc4297aed3b00df489f317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:44 UTC) from {username='slave'} at 127.0.0.1:41844
I20260621 07:55:44.604576 30410 raft_consensus.cc:2258] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:44.605511 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 29980
I20260621 07:55:44.606348 30410 ts_tablet_manager.cc:1905] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:55:44.609264 30410 ts_tablet_manager.cc:1918] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:55:44.609439 30410 log.cc:1196] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/79669eea51dc4297aed3b00df489f317
I20260621 07:55:44.609824 30410 ts_tablet_manager.cc:1939] T 79669eea51dc4297aed3b00df489f317 P 1016b6c99fdb4a068955f415458061b1: Deleting consensus metadata
I20260621 07:55:44.610105 30412 tablet_replica.cc:330] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc: stopping tablet replica
I20260621 07:55:44.611452 29905 catalog_manager.cc:4841] TS 1016b6c99fdb4a068955f415458061b1 (127.12.12.2:40345): tablet 79669eea51dc4297aed3b00df489f317 (table test-table [id=4a2b6929851e4eacbf6629a05b6ddb0e]) successfully deleted
I20260621 07:55:44.611681 30412 raft_consensus.cc:2229] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:55:44.612056 30412 raft_consensus.cc:2258] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:44.613667 30412 ts_tablet_manager.cc:1905] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260621 07:55:44.616288 29906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.1:45745 (error 108)
I20260621 07:55:44.616459 30412 ts_tablet_manager.cc:1918] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:55:44.616626 30412 log.cc:1196] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/79669eea51dc4297aed3b00df489f317
I20260621 07:55:44.616979 30412 ts_tablet_manager.cc:1939] T 79669eea51dc4297aed3b00df489f317 P 8b40b3fb2e2641939fe10561c34704fc: Deleting consensus metadata
I20260621 07:55:44.617573 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 30111
I20260621 07:55:44.617645 30385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:44.617350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:45745 (local address 127.0.0.1:50760)
0621 07:55:44.617482 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:44.617561 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:45745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260621 07:55:44.618690 29906 catalog_manager.cc:4584] TS 090008bd5c6243f1993d5bb78a82eb1f (127.12.12.1:45745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79669eea51dc4297aed3b00df489f317: Network error: Client connection negotiation failed: client connection to 127.12.12.1:45745: connect: Connection refused (error 111)
I20260621 07:55:44.618940 29906 catalog_manager.cc:4841] TS 8b40b3fb2e2641939fe10561c34704fc (127.12.12.3:41391): tablet 79669eea51dc4297aed3b00df489f317 (table test-table [id=4a2b6929851e4eacbf6629a05b6ddb0e]) successfully deleted
I20260621 07:55:44.624280 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 30242
I20260621 07:55:44.629931 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 29889
2026-06-21T07:55:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1231 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:55:44 dist-test-slave-lbpt krb5kdc[30416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:55:44 dist-test-slave-lbpt krb5kdc[30416](info): set up 2 sockets
Jun 21 07:55:44 dist-test-slave-lbpt krb5kdc[30416](info): commencing operation
krb5kdc: starting...
W20260621 07:55:45.682528 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:55:45 dist-test-slave-lbpt krb5kdc[30416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:55:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:55:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:55:45.803634 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:43261
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:43261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:45.917675 30429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:45.917963 30429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:45.918004 30429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:45.922124 30429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:55:45.922192 30429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:45.922212 30429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:45.922228 30429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:55:45.922243 30429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:55:45.927644 30429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:43261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:43261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:45.928769 30429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:45.929939 30429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:45.938592 30435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:45.938779 30429 server_base.cc:1034] running on GCE node
W20260621 07:55:45.938611 30437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:45.938611 30434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:45.939599 30429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:45.940773 30429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:45.942006 30429 hybrid_clock.cc:648] HybridClock initialized: now 1782028545941986 us; error 32 us; skew 500 ppm
Jun 21 07:55:45 dist-test-slave-lbpt krb5kdc[30416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028545, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:45.948046 30429 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:55:45.949043 30429 webserver.cc:460] Webserver started at http://127.12.12.62:43775/ using document root <none> and password file <none>
I20260621 07:55:45.949633 30429 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:45.949730 30429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:45.949980 30429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:45.952203 30429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "01e766ddb7b343899f31ac8577a4161f"
format_stamp: "Formatted at 2026-06-21 07:55:45 on dist-test-slave-lbpt"
server_key: "88ba13f73f04b5116d4b4b693e609e2c"
server_key_iv: "b1902c27d9231e59de9c70f3266a7d28"
server_key_version: "encryptionkey@0"
I20260621 07:55:45.952780 30429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "01e766ddb7b343899f31ac8577a4161f"
format_stamp: "Formatted at 2026-06-21 07:55:45 on dist-test-slave-lbpt"
server_key: "88ba13f73f04b5116d4b4b693e609e2c"
server_key_iv: "b1902c27d9231e59de9c70f3266a7d28"
server_key_version: "encryptionkey@0"
I20260621 07:55:45.956625 30429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260621 07:55:45.959456 30444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:45.960726 30429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:55:45.960894 30429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "01e766ddb7b343899f31ac8577a4161f"
format_stamp: "Formatted at 2026-06-21 07:55:45 on dist-test-slave-lbpt"
server_key: "88ba13f73f04b5116d4b4b693e609e2c"
server_key_iv: "b1902c27d9231e59de9c70f3266a7d28"
server_key_version: "encryptionkey@0"
I20260621 07:55:45.961019 30429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:45.979502 30429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:45.983811 30429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:45.984040 30429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:45.994222 30429 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:43261
I20260621 07:55:45.994257 30496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:43261 every 8 connection(s)
I20260621 07:55:45.995658 30429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:55:46.000141 30497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:46.000449 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 30429
I20260621 07:55:46.000597 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:55:46.000890 12336 external_mini_cluster.cc:1428] Setting key a29039dd152e9f3b47616143144ab406
I20260621 07:55:46.007593 30497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f: Bootstrap starting.
I20260621 07:55:46.010900 30497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f: Neither blocks nor log segments found. Creating new log.
Jun 21 07:55:46 dist-test-slave-lbpt krb5kdc[30416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:55:46.012075 30497 log.cc:824] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:46.014715 30497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f: No bootstrap required, opened a new log
I20260621 07:55:46.017313 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.002398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56132 (local address 127.12.12.62:43261)
0621 07:55:46.002897 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:46.002912 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:46.002949 (+ 37us) server_negotiation.cc:407] Connection header received
0621 07:55:46.003854 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:46.003873 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:46.004227 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:46.004575 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:46.005419 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.006565 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.007819 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.008486 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.009119 (+ 633us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.012350 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:46.012355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:46.012370 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:46.012401 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:46.014343 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:46.014894 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:46.014900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:46.014906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:46.015029 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:46.015325 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:46.015328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:46.015330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:46.015737 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:46.015935 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:46.016242 (+ 307us) server_negotiation.cc:299] Negotiation successful
0621 07:55:46.016570 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":153,"threads_started":1}
I20260621 07:55:46.018391 30497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e766ddb7b343899f31ac8577a4161f" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43261 } }
I20260621 07:55:46.018679 30497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:46.018729 30497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e766ddb7b343899f31ac8577a4161f, State: Initialized, Role: FOLLOWER
I20260621 07:55:46.019179 30497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e766ddb7b343899f31ac8577a4161f" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43261 } }
I20260621 07:55:46.019344 30497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:55:46.019407 30497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:55:46.019511 30497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:46.020463 30497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e766ddb7b343899f31ac8577a4161f" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43261 } }
I20260621 07:55:46.020746 30497 leader_election.cc:302] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01e766ddb7b343899f31ac8577a4161f; no voters:
I20260621 07:55:46.021020 30497 leader_election.cc:288] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:55:46.021157 30502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:46.021425 30502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [term 1 LEADER]: Becoming Leader. State: Replica: 01e766ddb7b343899f31ac8577a4161f, State: Running, Role: LEADER
I20260621 07:55:46.021770 30502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e766ddb7b343899f31ac8577a4161f" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43261 } }
I20260621 07:55:46.022276 30497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:55:46.023574 30503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01e766ddb7b343899f31ac8577a4161f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e766ddb7b343899f31ac8577a4161f" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43261 } } }
I20260621 07:55:46.023725 30503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:46.023574 30504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01e766ddb7b343899f31ac8577a4161f. Latest consensus state: current_term: 1 leader_uuid: "01e766ddb7b343899f31ac8577a4161f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e766ddb7b343899f31ac8577a4161f" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43261 } } }
I20260621 07:55:46.023962 30504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f [sys.catalog]: This master's current role is: LEADER
W20260621 07:55:46.027207 30517 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:55:46.027297 30517 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:55:46.027384 30512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:55:46.028077 30512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:55:46.033572 30512 catalog_manager.cc:1347] Generated new cluster ID: add4955fc5614d01ac29f7ef412b794a
I20260621 07:55:46.033641 30512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:55:46.045794 30512 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:55:46.046864 30512 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:55:46.060822 30512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f: Generated new TSK 0
I20260621 07:55:46.061640 30512 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:55:46.091183 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43261
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:46.211336 30523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:46.211634 30523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:46.211687 30523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:46.216058 30523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:46.216142 30523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:46.216204 30523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:55:46.221747 30523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:46.223104 30523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:46.224287 30523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:46.234623 30529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:46.234629 30531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:46.234948 30528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:46.234930 30523 server_base.cc:1034] running on GCE node
I20260621 07:55:46.235466 30523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:46.236316 30523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:46.237514 30523 hybrid_clock.cc:648] HybridClock initialized: now 1782028546237470 us; error 55 us; skew 500 ppm
Jun 21 07:55:46 dist-test-slave-lbpt krb5kdc[30416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028546, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:46.240761 30523 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:55:46.241726 30523 webserver.cc:460] Webserver started at http://127.12.12.1:36643/ using document root <none> and password file <none>
I20260621 07:55:46.242319 30523 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:46.242386 30523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:46.242640 30523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:46.244828 30523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "eb7bd8a8e2894698b3cc466c97446dd9"
format_stamp: "Formatted at 2026-06-21 07:55:46 on dist-test-slave-lbpt"
server_key: "02369d8115891f464d05aff67eab7f2e"
server_key_iv: "7207f5fb3e9f9fdf517482a874731dc9"
server_key_version: "encryptionkey@0"
I20260621 07:55:46.245406 30523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "eb7bd8a8e2894698b3cc466c97446dd9"
format_stamp: "Formatted at 2026-06-21 07:55:46 on dist-test-slave-lbpt"
server_key: "02369d8115891f464d05aff67eab7f2e"
server_key_iv: "7207f5fb3e9f9fdf517482a874731dc9"
server_key_version: "encryptionkey@0"
I20260621 07:55:46.249490 30523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 07:55:46.253108 30538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:46.254899 30523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 07:55:46.255048 30523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "eb7bd8a8e2894698b3cc466c97446dd9"
format_stamp: "Formatted at 2026-06-21 07:55:46 on dist-test-slave-lbpt"
server_key: "02369d8115891f464d05aff67eab7f2e"
server_key_iv: "7207f5fb3e9f9fdf517482a874731dc9"
server_key_version: "encryptionkey@0"
I20260621 07:55:46.255204 30523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:46.286660 30523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:46.290033 30523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:46.290206 30523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:46.290923 30523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:46.291772 30523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:46.291822 30523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:46.291863 30523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:46.291878 30523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:46.300879 30523 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:35645
I20260621 07:55:46.300925 30651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:35645 every 8 connection(s)
I20260621 07:55:46.301815 30523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:55:46.308290 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 30523
I20260621 07:55:46.308403 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:55:46.308708 12336 external_mini_cluster.cc:1428] Setting key 281cb7ab3fa3356c672f85dc54815504
Jun 21 07:55:46 dist-test-slave-lbpt krb5kdc[30416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028546, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:55:46.316859 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.303725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:33637 (local address 127.12.12.62:43261)
0621 07:55:46.303868 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:46.303871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:46.304940 (+ 1069us) server_negotiation.cc:407] Connection header received
0621 07:55:46.305856 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:46.305859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:46.305909 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:46.305994 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:46.307320 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.307877 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.309338 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.309626 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.309765 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.312616 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:46.312623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:46.312627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:46.312664 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:46.314445 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:46.314988 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:46.314995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:46.314998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:46.315091 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:46.315509 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:46.315515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:46.315520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:46.315808 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:46.315939 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:46.316459 (+ 520us) server_negotiation.cc:299] Negotiation successful
0621 07:55:46.316618 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":52}
I20260621 07:55:46.317698 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.304088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43261 (local address 127.12.12.1:33637)
0621 07:55:46.304757 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:46.304802 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:46.305607 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:46.306169 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:46.306176 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:46.306648 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:46.307150 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.307164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.308031 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.308034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.309198 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.309212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.309791 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.309795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.310011 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.310776 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:55:46.310793 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:46.312437 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:46.314588 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:46.314593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:46.314607 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:46.314882 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:46.315202 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:46.315205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:46.315207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:46.315406 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:46.315943 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:46.315948 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:46.316198 (+ 250us) client_negotiation.cc:769] Sending connection context
0621 07:55:46.316369 (+ 171us) client_negotiation.cc:240] Negotiation successful
0621 07:55:46.316643 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
I20260621 07:55:46.319137 30652 heartbeater.cc:347] Connected to a master server at 127.12.12.62:43261
I20260621 07:55:46.319410 30652 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:46.319937 30652 heartbeater.cc:511] Master 127.12.12.62:43261 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
I20260621 07:55:46.321693 30461 ts_manager.cc:194] Registered new tserver with Master: eb7bd8a8e2894698b3cc466c97446dd9 (127.12.12.1:35645)
I20260621 07:55:46.323248 30461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:33637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:55:46.337091 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43261
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:46.450919 30657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:46.451198 30657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:46.451241 30657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:46.455363 30657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:46.455428 30657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:46.455488 30657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:55:46.460955 30657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:46.462050 30657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:46.463151 30657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:46.473089 30665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:46.473098 30663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:46.473088 30662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:46.473610 30657 server_base.cc:1034] running on GCE node
I20260621 07:55:46.473977 30657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:46.474707 30657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:46.475869 30657 hybrid_clock.cc:648] HybridClock initialized: now 1782028546475831 us; error 51 us; skew 500 ppm
Jun 21 07:55:46 dist-test-slave-lbpt krb5kdc[30416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028546, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:46.478726 30657 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:55:46.479665 30657 webserver.cc:460] Webserver started at http://127.12.12.2:46375/ using document root <none> and password file <none>
I20260621 07:55:46.480258 30657 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:46.480321 30657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:46.480540 30657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:46.482690 30657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "d51337b71115458ebce868fdcbe1eb1b"
format_stamp: "Formatted at 2026-06-21 07:55:46 on dist-test-slave-lbpt"
server_key: "a5870707128bc6a15928791001b91474"
server_key_iv: "64cc920637870bfe82c14718a30611e1"
server_key_version: "encryptionkey@0"
I20260621 07:55:46.483258 30657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "d51337b71115458ebce868fdcbe1eb1b"
format_stamp: "Formatted at 2026-06-21 07:55:46 on dist-test-slave-lbpt"
server_key: "a5870707128bc6a15928791001b91474"
server_key_iv: "64cc920637870bfe82c14718a30611e1"
server_key_version: "encryptionkey@0"
I20260621 07:55:46.487006 30657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:55:46.489540 30672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:46.490899 30657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:55:46.491053 30657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "d51337b71115458ebce868fdcbe1eb1b"
format_stamp: "Formatted at 2026-06-21 07:55:46 on dist-test-slave-lbpt"
server_key: "a5870707128bc6a15928791001b91474"
server_key_iv: "64cc920637870bfe82c14718a30611e1"
server_key_version: "encryptionkey@0"
I20260621 07:55:46.491180 30657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:46.510250 30657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:46.513711 30657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:46.513927 30657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:46.514619 30657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:46.515470 30657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:46.515542 30657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:46.515617 30657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:46.515669 30657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:46.525334 30657 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:45501
I20260621 07:55:46.525349 30785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:45501 every 8 connection(s)
I20260621 07:55:46.526301 30657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:55:46.534073 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 30657
I20260621 07:55:46.534173 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:55:46.534531 12336 external_mini_cluster.cc:1428] Setting key 8fad2d2d38a1ec8b7302533a2b933e5e
Jun 21 07:55:46 dist-test-slave-lbpt krb5kdc[30416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028546, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:55:46.540361 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.528239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:41263 (local address 127.12.12.62:43261)
0621 07:55:46.528394 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:46.528400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:46.529478 (+ 1078us) server_negotiation.cc:407] Connection header received
0621 07:55:46.530342 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:46.530345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:46.530392 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:46.530525 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:46.531712 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.532227 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.533351 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.533562 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.533639 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.536424 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:46.536428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:46.536431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:46.536463 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:46.538112 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:46.538768 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:46.538771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:46.538772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:46.538854 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:46.539254 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:46.539256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:46.539257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:46.539432 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:46.539507 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:46.540128 (+ 621us) server_negotiation.cc:299] Negotiation successful
0621 07:55:46.540227 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 07:55:46.541666 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.528692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43261 (local address 127.12.12.2:41263)
0621 07:55:46.529331 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:46.529374 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:46.530128 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:46.530689 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:46.530696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:46.531098 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:46.531576 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.531589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.532339 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.532342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.533234 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.533244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.533700 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.533703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.533851 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.534558 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:55:46.534575 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:46.536254 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:46.538227 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:46.538245 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:46.538265 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:46.538668 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:46.538953 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:46.538957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:46.538959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:46.539158 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:46.539536 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:46.539546 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:46.539879 (+ 333us) client_negotiation.cc:769] Sending connection context
0621 07:55:46.540094 (+ 215us) client_negotiation.cc:240] Negotiation successful
0621 07:55:46.540474 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":113,"threads_started":1}
I20260621 07:55:46.542972 30786 heartbeater.cc:347] Connected to a master server at 127.12.12.62:43261
I20260621 07:55:46.543236 30786 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:46.543746 30786 heartbeater.cc:511] Master 127.12.12.62:43261 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
I20260621 07:55:46.544883 30461 ts_manager.cc:194] Registered new tserver with Master: d51337b71115458ebce868fdcbe1eb1b (127.12.12.2:45501)
I20260621 07:55:46.545506 30461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:41263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:55:46.561517 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43261
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:46.673669 30791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:46.674012 30791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:46.674093 30791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:46.678170 30791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:46.678274 30791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:46.678375 30791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:55:46.683755 30791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:46.684942 30791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:46.686022 30791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:46.696086 30797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:46.696094 30796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:46.696094 30799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:46.696372 30791 server_base.cc:1034] running on GCE node
I20260621 07:55:46.696934 30791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:46.697587 30791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:46.698777 30791 hybrid_clock.cc:648] HybridClock initialized: now 1782028546698754 us; error 52 us; skew 500 ppm
Jun 21 07:55:46 dist-test-slave-lbpt krb5kdc[30416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028546, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:46.701512 30791 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:55:46.702556 30791 webserver.cc:460] Webserver started at http://127.12.12.3:38711/ using document root <none> and password file <none>
I20260621 07:55:46.703138 30791 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:46.703229 30791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:46.703472 30791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:46.705601 30791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "138fc71b55214f5b910eb5323bfed8b1"
format_stamp: "Formatted at 2026-06-21 07:55:46 on dist-test-slave-lbpt"
server_key: "939ac46a5ab2a88632271043acdb2de3"
server_key_iv: "f75f9a60cc1d49e79d593bbb1f8a6b74"
server_key_version: "encryptionkey@0"
I20260621 07:55:46.706207 30791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "138fc71b55214f5b910eb5323bfed8b1"
format_stamp: "Formatted at 2026-06-21 07:55:46 on dist-test-slave-lbpt"
server_key: "939ac46a5ab2a88632271043acdb2de3"
server_key_iv: "f75f9a60cc1d49e79d593bbb1f8a6b74"
server_key_version: "encryptionkey@0"
I20260621 07:55:46.710498 30791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:55:46.713267 30806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:46.714758 30791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:55:46.714920 30791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "138fc71b55214f5b910eb5323bfed8b1"
format_stamp: "Formatted at 2026-06-21 07:55:46 on dist-test-slave-lbpt"
server_key: "939ac46a5ab2a88632271043acdb2de3"
server_key_iv: "f75f9a60cc1d49e79d593bbb1f8a6b74"
server_key_version: "encryptionkey@0"
I20260621 07:55:46.715052 30791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:46.739399 30791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:46.742646 30791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:46.742864 30791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:46.743479 30791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:46.744371 30791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:46.744441 30791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:46.744508 30791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:46.744555 30791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:46.753862 30791 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:35441
I20260621 07:55:46.753891 30919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:35441 every 8 connection(s)
I20260621 07:55:46.754845 30791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:55:46.759003 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 30791
I20260621 07:55:46.759150 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:55:46.759433 12336 external_mini_cluster.cc:1428] Setting key b9b0ee40709882ac180d3a6986f107c9
Jun 21 07:55:46 dist-test-slave-lbpt krb5kdc[30416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028546, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:55:46.769201 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.756721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:37949 (local address 127.12.12.62:43261)
0621 07:55:46.756891 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:46.756896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:46.758017 (+ 1121us) server_negotiation.cc:407] Connection header received
0621 07:55:46.759142 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:46.759146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:46.759204 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:46.759332 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:46.760946 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.761704 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.762959 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.763201 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.763315 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.765946 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:46.765949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:46.765951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:46.765976 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:46.767323 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:46.767802 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:46.767805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:46.767806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:46.767875 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:46.768184 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:46.768186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:46.768187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:46.768370 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:46.768446 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:46.768978 (+ 532us) server_negotiation.cc:299] Negotiation successful
0621 07:55:46.769080 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 07:55:46.770134 30922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.757181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43261 (local address 127.12.12.3:37949)
0621 07:55:46.757862 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:46.757905 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:46.758887 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:46.759531 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:46.759541 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:46.760106 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:46.760801 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.760822 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.761834 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.761837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.762839 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.762849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.763351 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.763355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.763538 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.764176 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:55:46.764203 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:46.765784 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:46.767430 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:46.767435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:46.767448 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:46.767710 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:46.767953 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:46.767956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:46.767957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:46.768108 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:46.768490 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:46.768495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:46.768750 (+ 255us) client_negotiation.cc:769] Sending connection context
0621 07:55:46.768925 (+ 175us) client_negotiation.cc:240] Negotiation successful
0621 07:55:46.769234 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":107,"threads_started":1}
I20260621 07:55:46.771430 30920 heartbeater.cc:347] Connected to a master server at 127.12.12.62:43261
I20260621 07:55:46.771651 30920 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:46.772229 30920 heartbeater.cc:511] Master 127.12.12.62:43261 requested a full tablet report, sending...
I20260621 07:55:46.773631 30460 ts_manager.cc:194] Registered new tserver with Master: 138fc71b55214f5b910eb5323bfed8b1 (127.12.12.3:35441)
I20260621 07:55:46.774257 30460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:37949
I20260621 07:55:46.785820 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:55:46.795715 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.787960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.12.12.62:43261)
0621 07:55:46.788132 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:46.788136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:46.788324 (+ 188us) server_negotiation.cc:407] Connection header received
0621 07:55:46.788408 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:46.788410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:46.788459 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:46.788535 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:46.789227 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.789773 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.790628 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.790895 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.790977 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.792502 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:46.792505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:46.792507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:46.792531 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:46.793828 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:46.794319 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:46.794322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:46.794323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:46.794487 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:46.794872 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:46.794875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:46.794876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:46.795090 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:46.795224 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:46.795465 (+ 241us) server_negotiation.cc:299] Negotiation successful
0621 07:55:46.795595 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:55:46.798916 30460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:55:46.801988 30460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:55:46.816599 30931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.810564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:35441 (local address 127.0.0.1:35258)
0621 07:55:46.810948 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:46.810995 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:46.811721 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:46.812519 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:46.812521 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:46.812553 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:46.812708 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.812714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.813908 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.813913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.815139 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.815148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.816017 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.816020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.816106 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.816137 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 07:55:46.816314 (+ 177us) client_negotiation.cc:240] Negotiation successful
0621 07:55:46.816442 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":21,"spinlock_wait_cycles":8704,"thread_start_us":112,"threads_started":1}
I20260621 07:55:46.816608 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.811436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35258 (local address 127.12.12.3:35441)
0621 07:55:46.811893 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:46.811912 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:46.811933 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:55:46.811996 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:46.812000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:46.812191 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:46.812340 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:46.812860 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.813726 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.815265 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.815876 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.815990 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.816144 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:46.816354 (+ 210us) server_negotiation.cc:299] Negotiation successful
0621 07:55:46.816458 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":138,"threads_started":1}
I20260621 07:55:46.817471 30932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.810602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:45501 (local address 127.0.0.1:34212)
0621 07:55:46.811619 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:46.811633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:46.811792 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:46.812514 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:46.812519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:46.812554 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:46.812737 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.812744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.814312 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.814317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.815737 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.815750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.817024 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.817027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.817138 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.817159 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:55:46.817234 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 07:55:46.817312 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":143,"spinlock_wait_cycles":4608,"thread_start_us":145,"threads_started":1}
I20260621 07:55:46.817736 30933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.811024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.12.12.2:45501)
0621 07:55:46.811456 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:46.811469 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:46.811715 (+ 246us) server_negotiation.cc:407] Connection header received
0621 07:55:46.811971 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:46.811977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:46.812191 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:46.812324 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:46.812876 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.814142 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.815876 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.816708 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.816817 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.816956 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:46.817301 (+ 345us) server_negotiation.cc:299] Negotiation successful
0621 07:55:46.817405 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":124,"threads_started":1}
I20260621 07:55:46.818056 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.811330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:35645 (local address 127.0.0.1:52616)
0621 07:55:46.812212 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:46.812222 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:46.812340 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:46.812874 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:46.812876 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:46.812892 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:46.812983 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.812989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.814343 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.814345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.815744 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.815749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.817324 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.817328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.817688 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.817777 (+ 89us) client_negotiation.cc:769] Sending connection context
0621 07:55:46.817854 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 07:55:46.817928 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":136,"thread_start_us":95,"threads_started":1}
I20260621 07:55:46.818548 30935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.811494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.12.12.1:35645)
0621 07:55:46.812085 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:46.812109 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:46.812318 (+ 209us) server_negotiation.cc:407] Connection header received
0621 07:55:46.812478 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:46.812486 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:46.812679 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:46.812855 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:46.813098 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.814213 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.816216 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.817034 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.817133 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.817268 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:46.818208 (+ 940us) server_negotiation.cc:299] Negotiation successful
0621 07:55:46.818311 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":76,"threads_started":1}
I20260621 07:55:46.820235 30854 tablet_service.cc:1467] Processing CreateTablet for tablet 61b1902fc373420eb60bf4c59b669f91 (DEFAULT_TABLE table=test-table [id=6eefa8afe7944bcbbb41564ae559dfaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:46.820369 30720 tablet_service.cc:1467] Processing CreateTablet for tablet 61b1902fc373420eb60bf4c59b669f91 (DEFAULT_TABLE table=test-table [id=6eefa8afe7944bcbbb41564ae559dfaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:46.821535 30720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b1902fc373420eb60bf4c59b669f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:46.821540 30854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b1902fc373420eb60bf4c59b669f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:46.822731 30586 tablet_service.cc:1467] Processing CreateTablet for tablet 61b1902fc373420eb60bf4c59b669f91 (DEFAULT_TABLE table=test-table [id=6eefa8afe7944bcbbb41564ae559dfaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:46.823750 30586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b1902fc373420eb60bf4c59b669f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:46.830560 30939 tablet_bootstrap.cc:492] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9: Bootstrap starting.
I20260621 07:55:46.830819 30938 tablet_bootstrap.cc:492] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1: Bootstrap starting.
I20260621 07:55:46.831139 30937 tablet_bootstrap.cc:492] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b: Bootstrap starting.
I20260621 07:55:46.832854 30938 tablet_bootstrap.cc:654] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:46.832963 30939 tablet_bootstrap.cc:654] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:46.833730 30937 tablet_bootstrap.cc:654] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:46.833967 30939 log.cc:824] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:46.834244 30938 log.cc:824] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:46.834676 30937 log.cc:824] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:46.836213 30938 tablet_bootstrap.cc:492] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1: No bootstrap required, opened a new log
I20260621 07:55:46.836422 30938 ts_tablet_manager.cc:1397] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:55:46.836443 30939 tablet_bootstrap.cc:492] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9: No bootstrap required, opened a new log
I20260621 07:55:46.836622 30939 ts_tablet_manager.cc:1397] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:55:46.837546 30937 tablet_bootstrap.cc:492] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b: No bootstrap required, opened a new log
I20260621 07:55:46.837729 30937 ts_tablet_manager.cc:1397] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260621 07:55:46.839715 30938 raft_consensus.cc:348] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138fc71b55214f5b910eb5323bfed8b1" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35441 } } peers { permanent_uuid: "d51337b71115458ebce868fdcbe1eb1b" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45501 } } peers { permanent_uuid: "eb7bd8a8e2894698b3cc466c97446dd9" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35645 } }
I20260621 07:55:46.840065 30938 raft_consensus.cc:374] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:46.840128 30938 raft_consensus.cc:729] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138fc71b55214f5b910eb5323bfed8b1, State: Initialized, Role: FOLLOWER
I20260621 07:55:46.840698 30938 consensus_queue.cc:260] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138fc71b55214f5b910eb5323bfed8b1" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35441 } } peers { permanent_uuid: "d51337b71115458ebce868fdcbe1eb1b" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45501 } } peers { permanent_uuid: "eb7bd8a8e2894698b3cc466c97446dd9" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35645 } }
I20260621 07:55:46.841413 30939 raft_consensus.cc:348] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138fc71b55214f5b910eb5323bfed8b1" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35441 } } peers { permanent_uuid: "d51337b71115458ebce868fdcbe1eb1b" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45501 } } peers { permanent_uuid: "eb7bd8a8e2894698b3cc466c97446dd9" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35645 } }
I20260621 07:55:46.841703 30939 raft_consensus.cc:374] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:46.841771 30939 raft_consensus.cc:729] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7bd8a8e2894698b3cc466c97446dd9, State: Initialized, Role: FOLLOWER
I20260621 07:55:46.841882 30938 ts_tablet_manager.cc:1428] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260621 07:55:46.841737 30937 raft_consensus.cc:348] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138fc71b55214f5b910eb5323bfed8b1" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35441 } } peers { permanent_uuid: "d51337b71115458ebce868fdcbe1eb1b" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45501 } } peers { permanent_uuid: "eb7bd8a8e2894698b3cc466c97446dd9" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35645 } }
I20260621 07:55:46.841989 30937 raft_consensus.cc:374] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:46.842056 30937 raft_consensus.cc:729] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d51337b71115458ebce868fdcbe1eb1b, State: Initialized, Role: FOLLOWER
I20260621 07:55:46.842224 30920 heartbeater.cc:503] Master 127.12.12.62:43261 was elected leader, sending a full tablet report...
I20260621 07:55:46.842382 30939 consensus_queue.cc:260] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138fc71b55214f5b910eb5323bfed8b1" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35441 } } peers { permanent_uuid: "d51337b71115458ebce868fdcbe1eb1b" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45501 } } peers { permanent_uuid: "eb7bd8a8e2894698b3cc466c97446dd9" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35645 } }
I20260621 07:55:46.842458 30937 consensus_queue.cc:260] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138fc71b55214f5b910eb5323bfed8b1" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35441 } } peers { permanent_uuid: "d51337b71115458ebce868fdcbe1eb1b" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45501 } } peers { permanent_uuid: "eb7bd8a8e2894698b3cc466c97446dd9" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35645 } }
I20260621 07:55:46.843369 30786 heartbeater.cc:503] Master 127.12.12.62:43261 was elected leader, sending a full tablet report...
I20260621 07:55:46.843503 30937 ts_tablet_manager.cc:1428] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260621 07:55:46.843708 30652 heartbeater.cc:503] Master 127.12.12.62:43261 was elected leader, sending a full tablet report...
I20260621 07:55:46.844190 30939 ts_tablet_manager.cc:1428] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260621 07:55:46.873016 30943 raft_consensus.cc:482] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:55:46.873281 30943 raft_consensus.cc:504] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138fc71b55214f5b910eb5323bfed8b1" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35441 } } peers { permanent_uuid: "d51337b71115458ebce868fdcbe1eb1b" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45501 } } peers { permanent_uuid: "eb7bd8a8e2894698b3cc466c97446dd9" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35645 } }
I20260621 07:55:46.874680 30943 leader_election.cc:288] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d51337b71115458ebce868fdcbe1eb1b (127.12.12.2:45501), eb7bd8a8e2894698b3cc466c97446dd9 (127.12.12.1:35645)
I20260621 07:55:46.876649 30944 raft_consensus.cc:482] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:55:46.876840 30944 raft_consensus.cc:504] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138fc71b55214f5b910eb5323bfed8b1" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35441 } } peers { permanent_uuid: "d51337b71115458ebce868fdcbe1eb1b" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45501 } } peers { permanent_uuid: "eb7bd8a8e2894698b3cc466c97446dd9" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35645 } }
I20260621 07:55:46.877869 30944 leader_election.cc:288] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 138fc71b55214f5b910eb5323bfed8b1 (127.12.12.3:35441), d51337b71115458ebce868fdcbe1eb1b (127.12.12.2:45501)
I20260621 07:55:46.879464 30922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.874859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:45501 (local address 127.12.12.3:50425)
0621 07:55:46.874967 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:46.874986 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:46.875079 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:46.875356 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:46.875359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:46.875378 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:46.875516 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.875525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.876569 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.876577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.877912 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.877924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.879004 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.879008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.879122 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.879145 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 07:55:46.879220 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 07:55:46.879308 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":78}
I20260621 07:55:46.879910 30935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.875174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:37725 (local address 127.12.12.1:35645)
0621 07:55:46.875331 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:46.875335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:46.875469 (+ 134us) server_negotiation.cc:407] Connection header received
0621 07:55:46.875606 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:46.875609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:46.875659 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:46.875737 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:46.875984 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.876661 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.878572 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.879156 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.879237 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.879334 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:46.879700 (+ 366us) server_negotiation.cc:299] Negotiation successful
0621 07:55:46.879776 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":54}
I20260621 07:55:46.880079 30933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.874954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:50425 (local address 127.12.12.2:45501)
0621 07:55:46.875093 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:46.875099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:46.875121 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:55:46.875180 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:46.875183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:46.875243 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:46.875343 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:46.875623 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.876444 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.878061 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.878793 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.878886 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.878949 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:46.879886 (+ 937us) server_negotiation.cc:299] Negotiation successful
0621 07:55:46.879971 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:55:46.880241 30946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.874948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:35645 (local address 127.12.12.3:37725)
0621 07:55:46.875367 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:46.875379 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:46.875490 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:46.875763 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:46.875765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:46.875779 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:46.875878 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.875882 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.876762 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.876765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.878435 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.878447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.879340 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.879343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.879446 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.879464 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 07:55:46.880022 (+ 558us) client_negotiation.cc:240] Negotiation successful
0621 07:55:46.880093 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":145,"threads_started":1}
I20260621 07:55:46.880597 30740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b1902fc373420eb60bf4c59b669f91" candidate_uuid: "138fc71b55214f5b910eb5323bfed8b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d51337b71115458ebce868fdcbe1eb1b" is_pre_election: true
I20260621 07:55:46.880813 30606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b1902fc373420eb60bf4c59b669f91" candidate_uuid: "138fc71b55214f5b910eb5323bfed8b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb7bd8a8e2894698b3cc466c97446dd9" is_pre_election: true
I20260621 07:55:46.880890 30740 raft_consensus.cc:2454] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 138fc71b55214f5b910eb5323bfed8b1 in term 0.
I20260621 07:55:46.881017 30606 raft_consensus.cc:2454] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 138fc71b55214f5b910eb5323bfed8b1 in term 0.
I20260621 07:55:46.881379 30809 leader_election.cc:302] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138fc71b55214f5b910eb5323bfed8b1, d51337b71115458ebce868fdcbe1eb1b; no voters:
I20260621 07:55:46.881649 30943 raft_consensus.cc:2781] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:55:46.881739 30943 raft_consensus.cc:482] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:55:46.881800 30943 raft_consensus.cc:3037] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:46.882647 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.878469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:39681 (local address 127.12.12.3:35441)
0621 07:55:46.878569 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:46.878573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:46.878587 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:55:46.878742 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:46.878748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:46.878799 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:46.878859 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:46.879286 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.880012 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.881595 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.882099 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.882169 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.882208 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:46.882485 (+ 277us) server_negotiation.cc:299] Negotiation successful
0621 07:55:46.882547 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 07:55:46.882684 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.878007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:35441 (local address 127.12.12.1:39681)
0621 07:55:46.878364 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:46.878381 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:46.878618 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:46.878992 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:46.878995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:46.879014 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:46.879160 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.879169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.880234 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.880237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.881458 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.881470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.882267 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.882269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.882341 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.882357 (+ 16us) client_negotiation.cc:769] Sending connection context
0621 07:55:46.882470 (+ 113us) client_negotiation.cc:240] Negotiation successful
0621 07:55:46.882540 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":32,"thread_start_us":129,"threads_started":1}
I20260621 07:55:46.883199 30874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b1902fc373420eb60bf4c59b669f91" candidate_uuid: "eb7bd8a8e2894698b3cc466c97446dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138fc71b55214f5b910eb5323bfed8b1" is_pre_election: true
I20260621 07:55:46.883216 30943 raft_consensus.cc:504] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138fc71b55214f5b910eb5323bfed8b1" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35441 } } peers { permanent_uuid: "d51337b71115458ebce868fdcbe1eb1b" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45501 } } peers { permanent_uuid: "eb7bd8a8e2894698b3cc466c97446dd9" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35645 } }
I20260621 07:55:46.883543 30874 raft_consensus.cc:2379] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb7bd8a8e2894698b3cc466c97446dd9 in current term 1: Already voted for candidate 138fc71b55214f5b910eb5323bfed8b1 in this term.
I20260621 07:55:46.883577 30947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.878087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:45501 (local address 127.12.12.1:37941)
0621 07:55:46.878624 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:46.878636 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:46.878721 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:46.879197 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:46.879203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:46.879221 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:46.879439 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.879445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.881035 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.881042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.882199 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:46.882209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.883156 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.883160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:46.883268 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.883292 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 07:55:46.883380 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 07:55:46.883440 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":29,"thread_start_us":108,"threads_started":1}
I20260621 07:55:46.883708 30943 leader_election.cc:288] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [CANDIDATE]: Term 1 election: Requested vote from peers d51337b71115458ebce868fdcbe1eb1b (127.12.12.2:45501), eb7bd8a8e2894698b3cc466c97446dd9 (127.12.12.1:35645)
I20260621 07:55:46.883927 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.878193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:37941 (local address 127.12.12.2:45501)
0621 07:55:46.878981 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:46.878990 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:46.879001 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:55:46.879039 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:46.879042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:46.879080 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:46.879539 (+ 459us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:46.879625 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.880905 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.882346 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.882983 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.883144 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.883214 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:46.883499 (+ 285us) server_negotiation.cc:299] Negotiation successful
0621 07:55:46.883572 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":107,"threads_started":1}
I20260621 07:55:46.884155 30740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b1902fc373420eb60bf4c59b669f91" candidate_uuid: "138fc71b55214f5b910eb5323bfed8b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d51337b71115458ebce868fdcbe1eb1b"
I20260621 07:55:46.884294 30740 raft_consensus.cc:3037] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:46.884330 30739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b1902fc373420eb60bf4c59b669f91" candidate_uuid: "eb7bd8a8e2894698b3cc466c97446dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d51337b71115458ebce868fdcbe1eb1b" is_pre_election: true
I20260621 07:55:46.884471 30606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b1902fc373420eb60bf4c59b669f91" candidate_uuid: "138fc71b55214f5b910eb5323bfed8b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb7bd8a8e2894698b3cc466c97446dd9"
I20260621 07:55:46.884598 30606 raft_consensus.cc:3037] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:46.884987 30541 leader_election.cc:302] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb7bd8a8e2894698b3cc466c97446dd9; no voters: 138fc71b55214f5b910eb5323bfed8b1, d51337b71115458ebce868fdcbe1eb1b
I20260621 07:55:46.885675 30740 raft_consensus.cc:2454] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 138fc71b55214f5b910eb5323bfed8b1 in term 1.
I20260621 07:55:46.885809 30606 raft_consensus.cc:2454] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 138fc71b55214f5b910eb5323bfed8b1 in term 1.
I20260621 07:55:46.885963 30944 raft_consensus.cc:2726] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260621 07:55:46.885988 30809 leader_election.cc:302] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138fc71b55214f5b910eb5323bfed8b1, d51337b71115458ebce868fdcbe1eb1b; no voters:
I20260621 07:55:46.886202 30943 raft_consensus.cc:2781] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:46.886536 30943 raft_consensus.cc:686] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [term 1 LEADER]: Becoming Leader. State: Replica: 138fc71b55214f5b910eb5323bfed8b1, State: Running, Role: LEADER
I20260621 07:55:46.886869 30943 consensus_queue.cc:237] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138fc71b55214f5b910eb5323bfed8b1" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35441 } } peers { permanent_uuid: "d51337b71115458ebce868fdcbe1eb1b" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45501 } } peers { permanent_uuid: "eb7bd8a8e2894698b3cc466c97446dd9" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35645 } }
I20260621 07:55:46.890154 30460 catalog_manager.cc:5495] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 138fc71b55214f5b910eb5323bfed8b1 (127.12.12.3). New cstate: current_term: 1 leader_uuid: "138fc71b55214f5b910eb5323bfed8b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138fc71b55214f5b910eb5323bfed8b1" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d51337b71115458ebce868fdcbe1eb1b" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb7bd8a8e2894698b3cc466c97446dd9" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35645 } health_report { overall_health: UNKNOWN } } }
I20260621 07:55:46.917971 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.914475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.12.12.3:35441)
0621 07:55:46.914620 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:46.914624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:46.914639 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:55:46.914767 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:46.914769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:46.914822 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:46.914938 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:55:46.915223 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.915715 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.916539 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:46.916746 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:46.916822 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:46.917081 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:55:46.917495 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:55:46.917605 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:46.917705 (+ 100us) server_negotiation.cc:299] Negotiation successful
0621 07:55:46.917738 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:55:46.924352 30606 raft_consensus.cc:1261] T 61b1902fc373420eb60bf4c59b669f91 P eb7bd8a8e2894698b3cc466c97446dd9 [term 1 FOLLOWER]: Refusing update from remote peer 138fc71b55214f5b910eb5323bfed8b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:55:46.924343 30740 raft_consensus.cc:1261] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b [term 1 FOLLOWER]: Refusing update from remote peer 138fc71b55214f5b910eb5323bfed8b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:55:46.925071 30950 consensus_queue.cc:1035] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d51337b71115458ebce868fdcbe1eb1b" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:55:46.925280 30943 consensus_queue.cc:1035] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb7bd8a8e2894698b3cc466c97446dd9" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 35645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:55:46.933940 30953 mvcc.cc:205] Tried to move back new op lower bound from 7299188928194052096 to 7299188928051662848. Current Snapshot: MvccSnapshot[applied={T|T < 7299188928194052096}]
I20260621 07:55:46.934130 30955 mvcc.cc:205] Tried to move back new op lower bound from 7299188928194052096 to 7299188928051662848. Current Snapshot: MvccSnapshot[applied={T|T < 7299188928194052096}]
I20260621 07:55:46.943084 30461 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:55:46.943244 30461 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:55:46.945726 30461 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01e766ddb7b343899f31ac8577a4161f: Sending DeleteTablet for 3 replicas of tablet 61b1902fc373420eb60bf4c59b669f91
I20260621 07:55:46.946635 30720 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b1902fc373420eb60bf4c59b669f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:46 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:34212
I20260621 07:55:46.946676 30854 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b1902fc373420eb60bf4c59b669f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:46 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:35258
I20260621 07:55:46.947242 30963 tablet_replica.cc:330] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b: stopping tablet replica
I20260621 07:55:46.947434 30964 tablet_replica.cc:330] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1: stopping tablet replica
I20260621 07:55:46.947561 30586 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b1902fc373420eb60bf4c59b669f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:46 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:52616
I20260621 07:55:46.947710 30964 raft_consensus.cc:2229] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:55:46.947824 30963 raft_consensus.cc:2229] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:55:46.948074 30964 raft_consensus.cc:2258] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:46.948170 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 30523
I20260621 07:55:46.948299 30963 raft_consensus.cc:2258] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:46.950834 30964 ts_tablet_manager.cc:1905] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:55:46.952283 30963 ts_tablet_manager.cc:1905] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:55:46.953791 30964 ts_tablet_manager.cc:1918] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:55:46.953907 30964 log.cc:1196] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/61b1902fc373420eb60bf4c59b669f91
I20260621 07:55:46.954274 30964 ts_tablet_manager.cc:1939] T 61b1902fc373420eb60bf4c59b669f91 P 138fc71b55214f5b910eb5323bfed8b1: Deleting consensus metadata
I20260621 07:55:46.955700 30963 ts_tablet_manager.cc:1918] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:55:46.955685 30445 catalog_manager.cc:4841] TS 138fc71b55214f5b910eb5323bfed8b1 (127.12.12.3:35441): tablet 61b1902fc373420eb60bf4c59b669f91 (table test-table [id=6eefa8afe7944bcbbb41564ae559dfaf]) successfully deleted
I20260621 07:55:46.955875 30963 log.cc:1196] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/61b1902fc373420eb60bf4c59b669f91
I20260621 07:55:46.956264 30963 ts_tablet_manager.cc:1939] T 61b1902fc373420eb60bf4c59b669f91 P d51337b71115458ebce868fdcbe1eb1b: Deleting consensus metadata
I20260621 07:55:46.957343 30447 catalog_manager.cc:4841] TS d51337b71115458ebce868fdcbe1eb1b (127.12.12.2:45501): tablet 61b1902fc373420eb60bf4c59b669f91 (table test-table [id=6eefa8afe7944bcbbb41564ae559dfaf]) successfully deleted
W20260621 07:55:46.957978 30445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.1:35645 (error 108)
I20260621 07:55:46.959080 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:46.958730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:35645 (local address 127.0.0.1:52624)
0621 07:55:46.958912 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:46.958972 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260621 07:55:46.959170 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 30657
W20260621 07:55:46.960390 30445 catalog_manager.cc:4584] TS eb7bd8a8e2894698b3cc466c97446dd9 (127.12.12.1:35645): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 61b1902fc373420eb60bf4c59b669f91: Network error: Client connection negotiation failed: client connection to 127.12.12.1:35645: connect: Connection refused (error 111)
I20260621 07:55:46.964725 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 30791
I20260621 07:55:46.970520 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 30429
2026-06-21T07:55:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2341 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-21T07:55:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:55:46Z Disabled control of system clock
I20260621 07:55:47.001159 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:33359
--webserver_interface=127.12.12.62
--webserver_port=0
--builtin_ntp_servers=127.12.12.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:33359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:55:47.116854 30970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:47.117141 30970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:47.117177 30970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:47.121272 30970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:55:47.121340 30970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:47.121356 30970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:47.121372 30970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:55:47.121387 30970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:55:47.126310 30970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:33359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:33359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:47.127530 30970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:47.128988 30970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:47.137853 30975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:47.137863 30976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:47.137887 30978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:47.138293 30970 server_base.cc:1034] running on GCE node
I20260621 07:55:47.138957 30970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:47.140077 30970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:47.141252 30970 hybrid_clock.cc:648] HybridClock initialized: now 1782028547141235 us; error 35 us; skew 500 ppm
I20260621 07:55:47.143366 30970 webserver.cc:460] Webserver started at http://127.12.12.62:44249/ using document root <none> and password file <none>
I20260621 07:55:47.143981 30970 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:47.144047 30970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:47.144281 30970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:47.146385 30970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "fee3e52cae32453598dcca37a725698a"
format_stamp: "Formatted at 2026-06-21 07:55:47 on dist-test-slave-lbpt"
server_key: "5216f03ae1e4bf3c0745eea968424305"
server_key_iv: "e180838b92fa48af37c494dc008587cc"
server_key_version: "encryptionkey@0"
I20260621 07:55:47.147033 30970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "fee3e52cae32453598dcca37a725698a"
format_stamp: "Formatted at 2026-06-21 07:55:47 on dist-test-slave-lbpt"
server_key: "5216f03ae1e4bf3c0745eea968424305"
server_key_iv: "e180838b92fa48af37c494dc008587cc"
server_key_version: "encryptionkey@0"
I20260621 07:55:47.150835 30970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:55:47.153296 30984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:47.154677 30970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:55:47.154785 30970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "fee3e52cae32453598dcca37a725698a"
format_stamp: "Formatted at 2026-06-21 07:55:47 on dist-test-slave-lbpt"
server_key: "5216f03ae1e4bf3c0745eea968424305"
server_key_iv: "e180838b92fa48af37c494dc008587cc"
server_key_version: "encryptionkey@0"
I20260621 07:55:47.154937 30970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:47.180316 30970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:47.181111 30970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:47.181339 30970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:47.188937 30970 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:33359
I20260621 07:55:47.188971 31036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:33359 every 8 connection(s)
I20260621 07:55:47.189949 30970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:55:47.193364 31037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:47.197875 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 30970
I20260621 07:55:47.198014 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:55:47.198305 12336 external_mini_cluster.cc:1428] Setting key 783cda10cbce95162d6fc4834268692f
I20260621 07:55:47.199765 31037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a: Bootstrap starting.
I20260621 07:55:47.202142 31037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:47.203047 31037 log.cc:824] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:47.204962 31037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a: No bootstrap required, opened a new log
I20260621 07:55:47.208258 31040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:47.199914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.12.12.62:33359)
0621 07:55:47.200628 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:47.200644 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:47.200683 (+ 39us) server_negotiation.cc:407] Connection header received
0621 07:55:47.201347 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:47.201366 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:47.201790 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:47.202164 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:47.202662 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.203799 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.204708 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.205256 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.205445 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:47.206347 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:47.206352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:47.206365 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:47.206835 (+ 470us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:47.206894 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:47.206916 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:47.207028 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:47.207185 (+ 157us) server_negotiation.cc:299] Negotiation successful
0621 07:55:47.207470 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":143,"threads_started":1}
I20260621 07:55:47.209120 31037 raft_consensus.cc:348] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee3e52cae32453598dcca37a725698a" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33359 } }
I20260621 07:55:47.209432 31037 raft_consensus.cc:374] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:47.209482 31037 raft_consensus.cc:729] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee3e52cae32453598dcca37a725698a, State: Initialized, Role: FOLLOWER
I20260621 07:55:47.209949 31037 consensus_queue.cc:260] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee3e52cae32453598dcca37a725698a" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33359 } }
I20260621 07:55:47.210084 31037 raft_consensus.cc:388] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:55:47.210177 31037 raft_consensus.cc:482] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:55:47.210304 31037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:47.211669 31037 raft_consensus.cc:504] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee3e52cae32453598dcca37a725698a" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33359 } }
I20260621 07:55:47.212056 31037 leader_election.cc:302] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fee3e52cae32453598dcca37a725698a; no voters:
I20260621 07:55:47.212443 31037 leader_election.cc:288] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:55:47.212623 31042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:47.212913 31042 raft_consensus.cc:686] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [term 1 LEADER]: Becoming Leader. State: Replica: fee3e52cae32453598dcca37a725698a, State: Running, Role: LEADER
I20260621 07:55:47.213236 31042 consensus_queue.cc:237] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee3e52cae32453598dcca37a725698a" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33359 } }
I20260621 07:55:47.213873 31037 sys_catalog.cc:564] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:55:47.214953 31043 sys_catalog.cc:455] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fee3e52cae32453598dcca37a725698a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee3e52cae32453598dcca37a725698a" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33359 } } }
I20260621 07:55:47.215106 31043 sys_catalog.cc:458] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:47.215461 31051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:55:47.215423 31044 sys_catalog.cc:455] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fee3e52cae32453598dcca37a725698a. Latest consensus state: current_term: 1 leader_uuid: "fee3e52cae32453598dcca37a725698a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee3e52cae32453598dcca37a725698a" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 33359 } } }
I20260621 07:55:47.215569 31044 sys_catalog.cc:458] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:47.219053 31051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:55:47.225157 31051 catalog_manager.cc:1347] Generated new cluster ID: da1d1b6bd47d4804aadcf3e905897f35
I20260621 07:55:47.225241 31051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:55:47.235926 31051 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:55:47.237289 31051 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:55:47.243708 31051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a: Generated new TSK 0
I20260621 07:55:47.244652 31051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:55:47.251561 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:33359
--builtin_ntp_servers=127.12.12.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:55:47.369624 31061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:47.369896 31061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:47.369974 31061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:47.374065 31061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:47.374130 31061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:47.374228 31061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:55:47.379069 31061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:33359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:47.380185 31061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:47.381600 31061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:47.391564 31069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:47.391569 31066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:47.391589 31067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:47.391821 31061 server_base.cc:1034] running on GCE node
I20260621 07:55:47.392530 31061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:47.393201 31061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:47.394469 31061 hybrid_clock.cc:648] HybridClock initialized: now 1782028547394400 us; error 28 us; skew 500 ppm
I20260621 07:55:47.396540 31061 webserver.cc:460] Webserver started at http://127.12.12.1:46059/ using document root <none> and password file <none>
I20260621 07:55:47.397130 31061 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:47.397203 31061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:47.397392 31061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:47.399673 31061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "37ef86a7c4d24a9db94a5ee20bab5320"
format_stamp: "Formatted at 2026-06-21 07:55:47 on dist-test-slave-lbpt"
server_key: "3a252ea45b02ca8b64137617542e2d55"
server_key_iv: "26da4539238d82cf43c786a6a790912f"
server_key_version: "encryptionkey@0"
I20260621 07:55:47.400209 31061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "37ef86a7c4d24a9db94a5ee20bab5320"
format_stamp: "Formatted at 2026-06-21 07:55:47 on dist-test-slave-lbpt"
server_key: "3a252ea45b02ca8b64137617542e2d55"
server_key_iv: "26da4539238d82cf43c786a6a790912f"
server_key_version: "encryptionkey@0"
I20260621 07:55:47.404100 31061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:55:47.406679 31075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:47.407974 31061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:55:47.408099 31061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "37ef86a7c4d24a9db94a5ee20bab5320"
format_stamp: "Formatted at 2026-06-21 07:55:47 on dist-test-slave-lbpt"
server_key: "3a252ea45b02ca8b64137617542e2d55"
server_key_iv: "26da4539238d82cf43c786a6a790912f"
server_key_version: "encryptionkey@0"
I20260621 07:55:47.408195 31061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:47.425159 31061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:47.425868 31061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:47.426023 31061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:47.426656 31061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:47.427610 31061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:47.427657 31061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:47.427697 31061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:47.427713 31061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:47.437031 31061 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:37199
I20260621 07:55:47.437098 31188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:37199 every 8 connection(s)
I20260621 07:55:47.438052 31061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:55:47.438722 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 31061
I20260621 07:55:47.438869 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:55:47.439162 12336 external_mini_cluster.cc:1428] Setting key 100f048e7128e0a14e395c3d7e04077f
I20260621 07:55:47.441254 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:33359
--builtin_ntp_servers=127.12.12.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:55:47.448447 31040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:47.439999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:49497 (local address 127.12.12.62:33359)
0621 07:55:47.440142 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:47.440147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:47.441508 (+ 1361us) server_negotiation.cc:407] Connection header received
0621 07:55:47.443358 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:47.443361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:47.443410 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:47.443487 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:47.444549 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.445114 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.446297 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.446619 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.446737 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:47.447933 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:47.447937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:47.447940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:47.447959 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:47.447980 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:47.447988 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:47.448065 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:47.448232 (+ 167us) server_negotiation.cc:299] Negotiation successful
0621 07:55:47.448297 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:55:47.449352 31191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:47.440555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33359 (local address 127.12.12.1:49497)
0621 07:55:47.441355 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:47.441411 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:47.443075 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:47.443727 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:47.443739 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:47.443871 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:47.444393 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:47.444407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.445255 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.445258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:47.446164 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:47.446173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.446781 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.446785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:47.446956 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:47.447666 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:55:47.447668 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:47.447709 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:55:47.447773 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:47.448078 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:47.448090 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:47.448098 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 07:55:47.448232 (+ 134us) client_negotiation.cc:240] Negotiation successful
0621 07:55:47.448532 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
I20260621 07:55:47.450730 31189 heartbeater.cc:347] Connected to a master server at 127.12.12.62:33359
I20260621 07:55:47.450997 31189 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:47.451522 31189 heartbeater.cc:511] Master 127.12.12.62:33359 requested a full tablet report, sending...
I20260621 07:55:47.453137 31001 ts_manager.cc:194] Registered new tserver with Master: 37ef86a7c4d24a9db94a5ee20bab5320 (127.12.12.1:37199)
I20260621 07:55:47.454789 31001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.12.1:49497
W20260621 07:55:47.557094 31192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:47.557363 31192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:47.557401 31192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:47.561353 31192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:47.561408 31192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:47.561527 31192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:55:47.566293 31192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:33359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:47.567502 31192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:47.568986 31192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:55:47.578948 31192 server_base.cc:1034] running on GCE node
W20260621 07:55:47.578860 31197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:47.578891 31198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:47.578860 31200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:47.579643 31192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:47.580339 31192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:47.581529 31192 hybrid_clock.cc:648] HybridClock initialized: now 1782028547581516 us; error 29 us; skew 500 ppm
I20260621 07:55:47.583469 31192 webserver.cc:460] Webserver started at http://127.12.12.2:41969/ using document root <none> and password file <none>
I20260621 07:55:47.584077 31192 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:47.584172 31192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:47.584417 31192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:47.586588 31192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "7be981e44aa3445f8a6ae342170916f4"
format_stamp: "Formatted at 2026-06-21 07:55:47 on dist-test-slave-lbpt"
server_key: "36f0409eb26a0b129ed28fea049b16ca"
server_key_iv: "d96909918899894bb43c8dbd77d00d1e"
server_key_version: "encryptionkey@0"
I20260621 07:55:47.587240 31192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "7be981e44aa3445f8a6ae342170916f4"
format_stamp: "Formatted at 2026-06-21 07:55:47 on dist-test-slave-lbpt"
server_key: "36f0409eb26a0b129ed28fea049b16ca"
server_key_iv: "d96909918899894bb43c8dbd77d00d1e"
server_key_version: "encryptionkey@0"
I20260621 07:55:47.591137 31192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:55:47.593832 31206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:47.595117 31192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:55:47.595247 31192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "7be981e44aa3445f8a6ae342170916f4"
format_stamp: "Formatted at 2026-06-21 07:55:47 on dist-test-slave-lbpt"
server_key: "36f0409eb26a0b129ed28fea049b16ca"
server_key_iv: "d96909918899894bb43c8dbd77d00d1e"
server_key_version: "encryptionkey@0"
I20260621 07:55:47.595367 31192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:47.612665 31192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:47.613433 31192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:47.613641 31192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:47.614317 31192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:47.615459 31192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:47.615506 31192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:47.615547 31192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:47.615563 31192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:47.624722 31192 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:44407
I20260621 07:55:47.624787 31319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:44407 every 8 connection(s)
I20260621 07:55:47.625744 31192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:55:47.628365 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 31192
I20260621 07:55:47.628496 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:55:47.628892 12336 external_mini_cluster.cc:1428] Setting key 1cda6ab498402138b4f8a5c02eb13ce0
I20260621 07:55:47.631201 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:33359
--builtin_ntp_servers=127.12.12.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:55:47.637657 31040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:47.627808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:33781 (local address 127.12.12.62:33359)
0621 07:55:47.627961 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:47.627965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:47.629441 (+ 1476us) server_negotiation.cc:407] Connection header received
0621 07:55:47.631199 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:47.631203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:47.631262 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:47.631365 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:47.632898 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.633775 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.635230 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.635673 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.635807 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:47.637031 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:47.637038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:47.637044 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:47.637063 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:47.637095 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:47.637109 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:47.637193 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:47.637353 (+ 160us) server_negotiation.cc:299] Negotiation successful
0621 07:55:47.637440 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 07:55:47.638545 31322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:47.628408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33359 (local address 127.12.12.2:33781)
0621 07:55:47.629237 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:47.629307 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:47.630959 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:47.631715 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:47.631728 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:47.631859 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:47.632697 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:47.632725 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.633927 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.633932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:47.635036 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:47.635048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.635889 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.635893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:47.636051 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:47.636742 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:55:47.636744 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:47.636787 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:55:47.636863 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:47.637205 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:47.637211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:47.637215 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 07:55:47.637316 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 07:55:47.637552 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":158,"threads_started":1}
I20260621 07:55:47.639951 31320 heartbeater.cc:347] Connected to a master server at 127.12.12.62:33359
I20260621 07:55:47.640242 31320 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:47.640844 31320 heartbeater.cc:511] Master 127.12.12.62:33359 requested a full tablet report, sending...
I20260621 07:55:47.642035 31001 ts_manager.cc:194] Registered new tserver with Master: 7be981e44aa3445f8a6ae342170916f4 (127.12.12.2:44407)
I20260621 07:55:47.642872 31001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.12.2:33781
W20260621 07:55:47.751958 31323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:47.752267 31323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:47.752310 31323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:47.756510 31323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:47.756589 31323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:47.756649 31323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:55:47.762070 31323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:33359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:47.763451 31323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:47.765213 31323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:47.775787 31328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:47.775794 31331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:47.775787 31329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:47.776122 31323 server_base.cc:1034] running on GCE node
I20260621 07:55:47.776620 31323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:47.777348 31323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:47.778517 31323 hybrid_clock.cc:648] HybridClock initialized: now 1782028547778470 us; error 53 us; skew 500 ppm
I20260621 07:55:47.780573 31323 webserver.cc:460] Webserver started at http://127.12.12.3:45587/ using document root <none> and password file <none>
I20260621 07:55:47.781184 31323 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:47.781276 31323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:47.781556 31323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:47.783740 31323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "40b55c14ebab49f4b62b1f0315c156ae"
format_stamp: "Formatted at 2026-06-21 07:55:47 on dist-test-slave-lbpt"
server_key: "ee922d08dba2d9810b5b38eaa103653b"
server_key_iv: "e85596541a4e1b3edcefc621c19a48ec"
server_key_version: "encryptionkey@0"
I20260621 07:55:47.784327 31323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "40b55c14ebab49f4b62b1f0315c156ae"
format_stamp: "Formatted at 2026-06-21 07:55:47 on dist-test-slave-lbpt"
server_key: "ee922d08dba2d9810b5b38eaa103653b"
server_key_iv: "e85596541a4e1b3edcefc621c19a48ec"
server_key_version: "encryptionkey@0"
I20260621 07:55:47.788247 31323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260621 07:55:47.790977 31337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:47.792285 31323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:55:47.792433 31323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "40b55c14ebab49f4b62b1f0315c156ae"
format_stamp: "Formatted at 2026-06-21 07:55:47 on dist-test-slave-lbpt"
server_key: "ee922d08dba2d9810b5b38eaa103653b"
server_key_iv: "e85596541a4e1b3edcefc621c19a48ec"
server_key_version: "encryptionkey@0"
I20260621 07:55:47.792551 31323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:47.811319 31323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:47.812101 31323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:47.812306 31323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:47.812948 31323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:47.813894 31323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:47.813964 31323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:47.814033 31323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:47.814090 31323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:47.823419 31323 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:46423
I20260621 07:55:47.823457 31450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:46423 every 8 connection(s)
I20260621 07:55:47.824365 31323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:55:47.828464 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 31323
I20260621 07:55:47.828576 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:55:47.828872 12336 external_mini_cluster.cc:1428] Setting key c4b80722f188f3ab217112c08b294f11
I20260621 07:55:47.834227 31040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:47.826277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:33805 (local address 127.12.12.62:33359)
0621 07:55:47.826449 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:47.826454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:47.827585 (+ 1131us) server_negotiation.cc:407] Connection header received
0621 07:55:47.829118 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:47.829122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:47.829186 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:47.829297 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:47.830301 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.830952 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.832168 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.832397 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.832475 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:47.833541 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:47.833544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:47.833546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:47.833557 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:47.833576 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:47.833584 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:47.833641 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:47.833977 (+ 336us) server_negotiation.cc:299] Negotiation successful
0621 07:55:47.834053 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:55:47.835269 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:47.826746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:33359 (local address 127.12.12.3:33805)
0621 07:55:47.827422 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:47.827463 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:47.828849 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:47.829492 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:47.829500 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:47.829625 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:47.830164 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:47.830176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.831095 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.831101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:47.832058 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:47.832068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.832525 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.832527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:47.832724 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:47.833297 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:55:47.833299 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:47.833343 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:55:47.833408 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:47.833736 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:47.833779 (+ 43us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:47.833789 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 07:55:47.833949 (+ 160us) client_negotiation.cc:240] Negotiation successful
0621 07:55:47.834342 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":112,"threads_started":1}
I20260621 07:55:47.836419 31451 heartbeater.cc:347] Connected to a master server at 127.12.12.62:33359
I20260621 07:55:47.836686 31451 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:47.837217 31451 heartbeater.cc:511] Master 127.12.12.62:33359 requested a full tablet report, sending...
I20260621 07:55:47.838238 31001 ts_manager.cc:194] Registered new tserver with Master: 40b55c14ebab49f4b62b1f0315c156ae (127.12.12.3:46423)
I20260621 07:55:47.838907 31001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.12.3:33805
I20260621 07:55:47.843109 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:55:47.849526 31040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:47.845160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.12.12.62:33359)
0621 07:55:47.845328 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:47.845333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:47.845522 (+ 189us) server_negotiation.cc:407] Connection header received
0621 07:55:47.845678 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:47.845681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:47.845728 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:47.845822 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:47.846126 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.846735 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.847590 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.847820 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.847938 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:47.848914 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:47.848926 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:47.848929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:47.848943 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:47.848964 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:47.848973 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:47.849138 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:47.849319 (+ 181us) server_negotiation.cc:299] Negotiation successful
0621 07:55:47.849364 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 07:55:47.853003 31001 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:55:47.856389 31001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:55:47.871604 31464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:47.865815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:46423 (local address 127.0.0.1:49542)
0621 07:55:47.866393 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:47.866405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:47.866513 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:47.867329 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:47.867336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:47.867372 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:47.867532 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:47.867539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.868757 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.868760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:47.870115 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:47.870125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.871058 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.871062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:47.871167 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:47.871202 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 07:55:47.871308 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 07:55:47.871430 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":35,"spinlock_wait_cycles":4864,"thread_start_us":110,"threads_started":1}
I20260621 07:55:47.871604 31466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:47.866444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.12.12.3:46423)
0621 07:55:47.866788 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:47.866800 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:47.866827 (+ 27us) server_negotiation.cc:407] Connection header received
0621 07:55:47.866911 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:47.866916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:47.867145 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:47.867685 (+ 540us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:47.867854 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.868626 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.870255 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.870925 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.871059 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:47.871280 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:47.871337 (+ 57us) server_negotiation.cc:299] Negotiation successful
0621 07:55:47.871430 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":111,"threads_started":1}
I20260621 07:55:47.871793 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:47.865498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:44407 (local address 127.0.0.1:37990)
0621 07:55:47.866081 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:47.866136 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:47.866303 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:47.867329 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:47.867336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:47.867373 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:47.867575 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:47.867587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.869111 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.869114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:47.870287 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:47.870294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.871387 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.871391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:47.871477 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:47.871500 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 07:55:47.871552 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 07:55:47.871624 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":81,"spinlock_wait_cycles":7168,"thread_start_us":113,"threads_started":1}
I20260621 07:55:47.871980 31465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:47.866266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:37199 (local address 127.0.0.1:35518)
0621 07:55:47.866700 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:47.866715 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:47.866805 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:47.867629 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:47.867632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:47.867649 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:47.867788 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:47.867795 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.869136 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.869138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:47.870257 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:47.870266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.871597 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.871601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:47.871686 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:47.871705 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 07:55:47.871785 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 07:55:47.871861 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":63,"thread_start_us":93,"threads_started":1}
I20260621 07:55:47.872396 31467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:47.866747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35518 (local address 127.12.12.1:37199)
0621 07:55:47.867165 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:47.867177 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:47.867201 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:55:47.867279 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:47.867286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:47.867494 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:47.867653 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:47.867910 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.869002 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.870392 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.871247 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.871364 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:47.871540 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:47.872140 (+ 600us) server_negotiation.cc:299] Negotiation successful
0621 07:55:47.872239 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":121,"threads_started":1}
I20260621 07:55:47.872699 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:47.865689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37990 (local address 127.12.12.2:44407)
0621 07:55:47.866142 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:47.866163 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:47.866910 (+ 747us) server_negotiation.cc:407] Connection header received
0621 07:55:47.866982 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:47.866986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:47.867145 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:47.867448 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:47.867730 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.868979 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.870451 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:47.871252 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:47.872207 (+ 955us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:47.872395 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:47.872447 (+ 52us) server_negotiation.cc:299] Negotiation successful
0621 07:55:47.872549 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":123,"threads_started":1}
I20260621 07:55:47.875044 31385 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0cce77e0fc4cd9bb5818530e239b5a (DEFAULT_TABLE table=test-table [id=376f5cb33ff14e09b3ff1b19b07036ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:47.875046 31254 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0cce77e0fc4cd9bb5818530e239b5a (DEFAULT_TABLE table=test-table [id=376f5cb33ff14e09b3ff1b19b07036ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:47.875985 31254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0cce77e0fc4cd9bb5818530e239b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:47.876015 31385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0cce77e0fc4cd9bb5818530e239b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:47.877456 31123 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0cce77e0fc4cd9bb5818530e239b5a (DEFAULT_TABLE table=test-table [id=376f5cb33ff14e09b3ff1b19b07036ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:47.878396 31123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0cce77e0fc4cd9bb5818530e239b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:47.882668 31468 tablet_bootstrap.cc:492] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4: Bootstrap starting.
I20260621 07:55:47.883927 31469 tablet_bootstrap.cc:492] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae: Bootstrap starting.
I20260621 07:55:47.884958 31470 tablet_bootstrap.cc:492] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320: Bootstrap starting.
I20260621 07:55:47.885040 31468 tablet_bootstrap.cc:654] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:47.885921 31468 log.cc:824] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:47.886045 31469 tablet_bootstrap.cc:654] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:47.886885 31469 log.cc:824] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:47.887213 31470 tablet_bootstrap.cc:654] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:47.888026 31470 log.cc:824] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:47.888756 31468 tablet_bootstrap.cc:492] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4: No bootstrap required, opened a new log
I20260621 07:55:47.888959 31468 ts_tablet_manager.cc:1397] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:55:47.889034 31469 tablet_bootstrap.cc:492] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae: No bootstrap required, opened a new log
I20260621 07:55:47.889207 31469 ts_tablet_manager.cc:1397] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:55:47.890070 31470 tablet_bootstrap.cc:492] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320: No bootstrap required, opened a new log
I20260621 07:55:47.890256 31470 ts_tablet_manager.cc:1397] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:55:47.893108 31470 raft_consensus.cc:348] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be981e44aa3445f8a6ae342170916f4" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44407 } } peers { permanent_uuid: "40b55c14ebab49f4b62b1f0315c156ae" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 46423 } } peers { permanent_uuid: "37ef86a7c4d24a9db94a5ee20bab5320" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 37199 } }
I20260621 07:55:47.893150 31468 raft_consensus.cc:348] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be981e44aa3445f8a6ae342170916f4" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44407 } } peers { permanent_uuid: "40b55c14ebab49f4b62b1f0315c156ae" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 46423 } } peers { permanent_uuid: "37ef86a7c4d24a9db94a5ee20bab5320" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 37199 } }
I20260621 07:55:47.893252 31469 raft_consensus.cc:348] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be981e44aa3445f8a6ae342170916f4" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44407 } } peers { permanent_uuid: "40b55c14ebab49f4b62b1f0315c156ae" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 46423 } } peers { permanent_uuid: "37ef86a7c4d24a9db94a5ee20bab5320" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 37199 } }
I20260621 07:55:47.893412 31468 raft_consensus.cc:374] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:47.893419 31470 raft_consensus.cc:374] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:47.893450 31469 raft_consensus.cc:374] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:47.893463 31468 raft_consensus.cc:729] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7be981e44aa3445f8a6ae342170916f4, State: Initialized, Role: FOLLOWER
I20260621 07:55:47.893476 31470 raft_consensus.cc:729] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ef86a7c4d24a9db94a5ee20bab5320, State: Initialized, Role: FOLLOWER
I20260621 07:55:47.893497 31469 raft_consensus.cc:729] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b55c14ebab49f4b62b1f0315c156ae, State: Initialized, Role: FOLLOWER
I20260621 07:55:47.893962 31470 consensus_queue.cc:260] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be981e44aa3445f8a6ae342170916f4" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44407 } } peers { permanent_uuid: "40b55c14ebab49f4b62b1f0315c156ae" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 46423 } } peers { permanent_uuid: "37ef86a7c4d24a9db94a5ee20bab5320" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 37199 } }
I20260621 07:55:47.893949 31469 consensus_queue.cc:260] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be981e44aa3445f8a6ae342170916f4" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44407 } } peers { permanent_uuid: "40b55c14ebab49f4b62b1f0315c156ae" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 46423 } } peers { permanent_uuid: "37ef86a7c4d24a9db94a5ee20bab5320" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 37199 } }
I20260621 07:55:47.893949 31468 consensus_queue.cc:260] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be981e44aa3445f8a6ae342170916f4" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44407 } } peers { permanent_uuid: "40b55c14ebab49f4b62b1f0315c156ae" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 46423 } } peers { permanent_uuid: "37ef86a7c4d24a9db94a5ee20bab5320" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 37199 } }
I20260621 07:55:47.894997 31470 ts_tablet_manager.cc:1428] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260621 07:55:47.894999 31469 ts_tablet_manager.cc:1428] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260621 07:55:47.894999 31468 ts_tablet_manager.cc:1428] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260621 07:55:47.895341 31451 heartbeater.cc:503] Master 127.12.12.62:33359 was elected leader, sending a full tablet report...
I20260621 07:55:47.895341 31320 heartbeater.cc:503] Master 127.12.12.62:33359 was elected leader, sending a full tablet report...
I20260621 07:55:47.895344 31189 heartbeater.cc:503] Master 127.12.12.62:33359 was elected leader, sending a full tablet report...
W20260621 07:55:47.939519 31190 tablet.cc:2369] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:55:48.023916 31476 raft_consensus.cc:482] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:55:48.024195 31476 raft_consensus.cc:504] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be981e44aa3445f8a6ae342170916f4" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44407 } } peers { permanent_uuid: "40b55c14ebab49f4b62b1f0315c156ae" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 46423 } } peers { permanent_uuid: "37ef86a7c4d24a9db94a5ee20bab5320" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 37199 } }
I20260621 07:55:48.025384 31476 leader_election.cc:288] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7be981e44aa3445f8a6ae342170916f4 (127.12.12.2:44407), 40b55c14ebab49f4b62b1f0315c156ae (127.12.12.3:46423)
I20260621 07:55:48.029439 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:48.025818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:47075 (local address 127.12.12.2:44407)
0621 07:55:48.025952 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:48.025958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:48.026172 (+ 214us) server_negotiation.cc:407] Connection header received
0621 07:55:48.026360 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:48.026362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:48.026462 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:48.026552 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:48.026878 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:48.027387 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:48.028477 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:48.028856 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:48.028973 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:48.029034 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:48.029229 (+ 195us) server_negotiation.cc:299] Negotiation successful
0621 07:55:48.029301 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:55:48.029452 31466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:48.025810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:59707 (local address 127.12.12.3:46423)
0621 07:55:48.025952 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:48.025957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:48.026294 (+ 337us) server_negotiation.cc:407] Connection header received
0621 07:55:48.026479 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:48.026482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:48.026533 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:48.026620 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:48.026936 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:48.027429 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:48.028483 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:48.028892 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:48.029012 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:48.029068 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:48.029255 (+ 187us) server_negotiation.cc:299] Negotiation successful
0621 07:55:48.029323 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:55:48.029489 31478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:48.025684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:46423 (local address 127.12.12.1:59707)
0621 07:55:48.026193 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:48.026205 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:48.026352 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:48.026634 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:48.026636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:48.026652 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:48.026815 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:48.026820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:48.027542 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:48.027546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:48.028357 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:48.028366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:48.029018 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:48.029021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:48.029121 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:48.029143 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:55:48.029238 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 07:55:48.029316 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":142,"thread_start_us":80,"threads_started":1}
I20260621 07:55:48.029513 31477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:48.025684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:44407 (local address 127.12.12.1:47075)
0621 07:55:48.026075 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:48.026088 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:48.026240 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:48.026575 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:48.026578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:48.026594 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:48.026752 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:48.026758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:48.027497 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:48.027502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:48.028352 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:48.028361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:48.028989 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:48.028993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:48.029097 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:48.029118 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:55:48.029207 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 07:55:48.029295 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":32,"thread_start_us":88,"threads_started":1}
I20260621 07:55:48.030285 31274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0cce77e0fc4cd9bb5818530e239b5a" candidate_uuid: "37ef86a7c4d24a9db94a5ee20bab5320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7be981e44aa3445f8a6ae342170916f4" is_pre_election: true
I20260621 07:55:48.030285 31405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0cce77e0fc4cd9bb5818530e239b5a" candidate_uuid: "37ef86a7c4d24a9db94a5ee20bab5320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b55c14ebab49f4b62b1f0315c156ae" is_pre_election: true
I20260621 07:55:48.030630 31274 raft_consensus.cc:2454] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ef86a7c4d24a9db94a5ee20bab5320 in term 0.
I20260621 07:55:48.030650 31405 raft_consensus.cc:2454] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ef86a7c4d24a9db94a5ee20bab5320 in term 0.
I20260621 07:55:48.031162 31078 leader_election.cc:302] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 37ef86a7c4d24a9db94a5ee20bab5320, 40b55c14ebab49f4b62b1f0315c156ae, 7be981e44aa3445f8a6ae342170916f4; no voters:
I20260621 07:55:48.031461 31476 raft_consensus.cc:2781] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:55:48.031535 31476 raft_consensus.cc:482] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:55:48.031566 31476 raft_consensus.cc:3037] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:48.032642 31476 raft_consensus.cc:504] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be981e44aa3445f8a6ae342170916f4" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44407 } } peers { permanent_uuid: "40b55c14ebab49f4b62b1f0315c156ae" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 46423 } } peers { permanent_uuid: "37ef86a7c4d24a9db94a5ee20bab5320" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 37199 } }
I20260621 07:55:48.033037 31476 leader_election.cc:288] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [CANDIDATE]: Term 1 election: Requested vote from peers 7be981e44aa3445f8a6ae342170916f4 (127.12.12.2:44407), 40b55c14ebab49f4b62b1f0315c156ae (127.12.12.3:46423)
I20260621 07:55:48.033422 31274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0cce77e0fc4cd9bb5818530e239b5a" candidate_uuid: "37ef86a7c4d24a9db94a5ee20bab5320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7be981e44aa3445f8a6ae342170916f4"
I20260621 07:55:48.033458 31405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0cce77e0fc4cd9bb5818530e239b5a" candidate_uuid: "37ef86a7c4d24a9db94a5ee20bab5320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b55c14ebab49f4b62b1f0315c156ae"
I20260621 07:55:48.033529 31274 raft_consensus.cc:3037] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:48.033605 31405 raft_consensus.cc:3037] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:48.034690 31274 raft_consensus.cc:2454] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ef86a7c4d24a9db94a5ee20bab5320 in term 1.
I20260621 07:55:48.034849 31405 raft_consensus.cc:2454] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ef86a7c4d24a9db94a5ee20bab5320 in term 1.
I20260621 07:55:48.035036 31078 leader_election.cc:302] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ef86a7c4d24a9db94a5ee20bab5320, 7be981e44aa3445f8a6ae342170916f4; no voters:
I20260621 07:55:48.035270 31476 raft_consensus.cc:2781] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:48.035511 31476 raft_consensus.cc:686] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [term 1 LEADER]: Becoming Leader. State: Replica: 37ef86a7c4d24a9db94a5ee20bab5320, State: Running, Role: LEADER
I20260621 07:55:48.035894 31476 consensus_queue.cc:237] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be981e44aa3445f8a6ae342170916f4" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44407 } } peers { permanent_uuid: "40b55c14ebab49f4b62b1f0315c156ae" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 46423 } } peers { permanent_uuid: "37ef86a7c4d24a9db94a5ee20bab5320" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 37199 } }
I20260621 07:55:48.039588 30997 catalog_manager.cc:5495] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37ef86a7c4d24a9db94a5ee20bab5320 (127.12.12.1). New cstate: current_term: 1 leader_uuid: "37ef86a7c4d24a9db94a5ee20bab5320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be981e44aa3445f8a6ae342170916f4" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40b55c14ebab49f4b62b1f0315c156ae" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 46423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37ef86a7c4d24a9db94a5ee20bab5320" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 37199 } health_report { overall_health: HEALTHY } } }
I20260621 07:55:48.063421 31467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:48.059630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.12.12.1:37199)
0621 07:55:48.059806 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:48.059810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:48.059824 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:55:48.059900 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:48.059902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:48.059960 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:48.060054 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:55:48.060347 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:48.060873 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:48.061755 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:48.061998 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:48.062079 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:48.062394 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:55:48.062966 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:55:48.063089 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:48.063268 (+ 179us) server_negotiation.cc:299] Negotiation successful
0621 07:55:48.063304 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 07:55:48.069793 31405 raft_consensus.cc:1261] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae [term 1 FOLLOWER]: Refusing update from remote peer 37ef86a7c4d24a9db94a5ee20bab5320: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:55:48.069793 31274 raft_consensus.cc:1261] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4 [term 1 FOLLOWER]: Refusing update from remote peer 37ef86a7c4d24a9db94a5ee20bab5320: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:55:48.070536 31479 consensus_queue.cc:1035] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7be981e44aa3445f8a6ae342170916f4" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:55:48.070757 31476 consensus_queue.cc:1035] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40b55c14ebab49f4b62b1f0315c156ae" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 46423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 07:55:48.075227 31452 tablet.cc:2369] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:55:48.081067 31484 mvcc.cc:205] Tried to move back new op lower bound from 7299188932886093824 to 7299188932757622784. Current Snapshot: MvccSnapshot[applied={T|T < 7299188932886093824}]
I20260621 07:55:48.086928 30997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:55:48.087080 30997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:55:48.089531 30997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fee3e52cae32453598dcca37a725698a: Sending DeleteTablet for 3 replicas of tablet 2e0cce77e0fc4cd9bb5818530e239b5a
I20260621 07:55:48.090343 31385 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0cce77e0fc4cd9bb5818530e239b5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:48 UTC) from {username='slave'} at 127.0.0.1:49542
I20260621 07:55:48.090521 31254 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0cce77e0fc4cd9bb5818530e239b5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:48 UTC) from {username='slave'} at 127.0.0.1:37990
I20260621 07:55:48.090927 31123 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0cce77e0fc4cd9bb5818530e239b5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:48 UTC) from {username='slave'} at 127.0.0.1:35518
I20260621 07:55:48.091095 31494 tablet_replica.cc:330] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4: stopping tablet replica
I20260621 07:55:48.091365 31495 tablet_replica.cc:330] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320: stopping tablet replica
I20260621 07:55:48.091430 31494 raft_consensus.cc:2229] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:55:48.091760 31495 raft_consensus.cc:2229] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:55:48.091781 31494 raft_consensus.cc:2258] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:48.092195 31495 raft_consensus.cc:2258] T 2e0cce77e0fc4cd9bb5818530e239b5a P 37ef86a7c4d24a9db94a5ee20bab5320 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:48.092226 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 31061
I20260621 07:55:48.093868 31494 ts_tablet_manager.cc:1905] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:55:48.096801 31494 ts_tablet_manager.cc:1918] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:55:48.096956 31494 log.cc:1196] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/2e0cce77e0fc4cd9bb5818530e239b5a
I20260621 07:55:48.097239 31494 ts_tablet_manager.cc:1939] T 2e0cce77e0fc4cd9bb5818530e239b5a P 7be981e44aa3445f8a6ae342170916f4: Deleting consensus metadata
I20260621 07:55:48.098443 30987 catalog_manager.cc:4841] TS 7be981e44aa3445f8a6ae342170916f4 (127.12.12.2:44407): tablet 2e0cce77e0fc4cd9bb5818530e239b5a (table test-table [id=376f5cb33ff14e09b3ff1b19b07036ba]) successfully deleted
I20260621 07:55:48.098713 31493 tablet_replica.cc:330] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae: stopping tablet replica
I20260621 07:55:48.099025 31493 raft_consensus.cc:2229] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:55:48.099435 31493 raft_consensus.cc:2258] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae [term 1 FOLLOWER]: Raft consensus is shut down!
W20260621 07:55:48.100395 30987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.1:37199 (error 108)
I20260621 07:55:48.100594 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 31192
I20260621 07:55:48.101451 31493 ts_tablet_manager.cc:1905] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:55:48.102239 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:48.101881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:37199 (local address 127.0.0.1:35530)
0621 07:55:48.102049 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:48.102132 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 07:55:48.102487 30987 catalog_manager.cc:4584] TS 37ef86a7c4d24a9db94a5ee20bab5320 (127.12.12.1:37199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e0cce77e0fc4cd9bb5818530e239b5a: Network error: Client connection negotiation failed: client connection to 127.12.12.1:37199: connect: Connection refused (error 111)
I20260621 07:55:48.103907 31493 ts_tablet_manager.cc:1918] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:55:48.104027 31493 log.cc:1196] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/2e0cce77e0fc4cd9bb5818530e239b5a
I20260621 07:55:48.104308 31493 ts_tablet_manager.cc:1939] T 2e0cce77e0fc4cd9bb5818530e239b5a P 40b55c14ebab49f4b62b1f0315c156ae: Deleting consensus metadata
I20260621 07:55:48.105536 30988 catalog_manager.cc:4841] TS 40b55c14ebab49f4b62b1f0315c156ae (127.12.12.3:46423): tablet 2e0cce77e0fc4cd9bb5818530e239b5a (table test-table [id=376f5cb33ff14e09b3ff1b19b07036ba]) successfully deleted
I20260621 07:55:48.107344 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 31323
I20260621 07:55:48.112870 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 30970
2026-06-21T07:55:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1141 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:55:48 dist-test-slave-lbpt krb5kdc[31499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:55:48 dist-test-slave-lbpt krb5kdc[31499](info): set up 2 sockets
Jun 21 07:55:48 dist-test-slave-lbpt krb5kdc[31499](info): commencing operation
krb5kdc: starting...
W20260621 07:55:49.158560 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:55:49 dist-test-slave-lbpt krb5kdc[31499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:55:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:55:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:55:49.283717 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:44023
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:44023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:49.401540 31512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:49.401893 31512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:49.401974 31512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:49.406203 31512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:55:49.406325 31512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:49.406363 31512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:49.406401 31512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:55:49.406474 31512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:55:49.412212 31512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:44023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:44023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:49.413385 31512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:49.414531 31512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:49.423285 31518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:49.423394 31512 server_base.cc:1034] running on GCE node
W20260621 07:55:49.423247 31520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:49.423651 31517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:49.424268 31512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:49.425410 31512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:49.426589 31512 hybrid_clock.cc:648] HybridClock initialized: now 1782028549426559 us; error 42 us; skew 500 ppm
Jun 21 07:55:49 dist-test-slave-lbpt krb5kdc[31499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028549, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:49.430220 31512 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:55:49.431479 31512 webserver.cc:460] Webserver started at http://127.12.12.62:38481/ using document root <none> and password file <none>
I20260621 07:55:49.432029 31512 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:49.432089 31512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:49.432252 31512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:49.434271 31512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "9572ef67eb1248449b516f6884680c4b"
format_stamp: "Formatted at 2026-06-21 07:55:49 on dist-test-slave-lbpt"
server_key: "7b2be9289c4548858d0282a37cb6aaf3"
server_key_iv: "875655e1216ca3d8d4e1063dd2b4b37a"
server_key_version: "encryptionkey@0"
I20260621 07:55:49.434870 31512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "9572ef67eb1248449b516f6884680c4b"
format_stamp: "Formatted at 2026-06-21 07:55:49 on dist-test-slave-lbpt"
server_key: "7b2be9289c4548858d0282a37cb6aaf3"
server_key_iv: "875655e1216ca3d8d4e1063dd2b4b37a"
server_key_version: "encryptionkey@0"
I20260621 07:55:49.438599 31512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 07:55:49.441689 31527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:49.443190 31512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:55:49.443353 31512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "9572ef67eb1248449b516f6884680c4b"
format_stamp: "Formatted at 2026-06-21 07:55:49 on dist-test-slave-lbpt"
server_key: "7b2be9289c4548858d0282a37cb6aaf3"
server_key_iv: "875655e1216ca3d8d4e1063dd2b4b37a"
server_key_version: "encryptionkey@0"
I20260621 07:55:49.443503 31512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:49.459251 31512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:49.462864 31512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:49.463086 31512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:49.470249 31512 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:44023
I20260621 07:55:49.470283 31579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:44023 every 8 connection(s)
I20260621 07:55:49.471342 31512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:55:49.474104 31580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:49.479751 31580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b: Bootstrap starting.
I20260621 07:55:49.481034 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 31512
I20260621 07:55:49.481170 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:55:49.481474 12336 external_mini_cluster.cc:1428] Setting key 5101c302b66f62afa728a889569c80d9
I20260621 07:55:49.482743 31580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:49.483549 31580 log.cc:824] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:49.485530 31580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b: No bootstrap required, opened a new log
I20260621 07:55:49.489043 31580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9572ef67eb1248449b516f6884680c4b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44023 } }
I20260621 07:55:49.489305 31580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:49.489390 31580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9572ef67eb1248449b516f6884680c4b, State: Initialized, Role: FOLLOWER
I20260621 07:55:49.489893 31580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9572ef67eb1248449b516f6884680c4b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44023 } }
I20260621 07:55:49.490046 31580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:55:49.490118 31580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:55:49.490231 31580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [term 0 FOLLOWER]: Advancing to term 1
Jun 21 07:55:49 dist-test-slave-lbpt krb5kdc[31499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:55:49.491562 31580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9572ef67eb1248449b516f6884680c4b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44023 } }
I20260621 07:55:49.491945 31580 leader_election.cc:302] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9572ef67eb1248449b516f6884680c4b; no voters:
I20260621 07:55:49.492329 31580 leader_election.cc:288] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:55:49.492453 31585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:49.492784 31585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [term 1 LEADER]: Becoming Leader. State: Replica: 9572ef67eb1248449b516f6884680c4b, State: Running, Role: LEADER
I20260621 07:55:49.493199 31585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9572ef67eb1248449b516f6884680c4b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44023 } }
I20260621 07:55:49.493623 31580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:55:49.495014 31586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9572ef67eb1248449b516f6884680c4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9572ef67eb1248449b516f6884680c4b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44023 } } }
I20260621 07:55:49.495239 31586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:49.495725 31596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:55:49.498005 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:49.482705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.12.12.62:44023)
0621 07:55:49.483167 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:49.483186 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:49.483245 (+ 59us) server_negotiation.cc:407] Connection header received
0621 07:55:49.484026 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:49.484050 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:49.484478 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:49.484829 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:49.485664 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:49.486720 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:49.487618 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:49.488147 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:49.488339 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:49.491395 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:49.491400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:49.491415 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:49.491444 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:49.493771 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:49.494380 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:49.494386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:49.494392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:49.494589 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:49.494938 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:49.494942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:49.494943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:49.495494 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:49.495692 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:49.496719 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0621 07:55:49.497038 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":147,"threads_started":1}
I20260621 07:55:49.498613 31587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9572ef67eb1248449b516f6884680c4b. Latest consensus state: current_term: 1 leader_uuid: "9572ef67eb1248449b516f6884680c4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9572ef67eb1248449b516f6884680c4b" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44023 } } }
I20260621 07:55:49.498723 31587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:49.499591 31596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:55:49.505281 31596 catalog_manager.cc:1347] Generated new cluster ID: 560e284a15184ec9b3156bb447e10195
I20260621 07:55:49.505373 31596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:55:49.525993 31596 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:55:49.527025 31596 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:55:49.532786 31596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b: Generated new TSK 0
I20260621 07:55:49.533502 31596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
Entry for principal kudu/127.12.12.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab.
I20260621 07:55:49.563788 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:44023
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:49.688568 31606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:49.688854 31606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:49.688899 31606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:49.693183 31606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:49.693253 31606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:49.693358 31606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:55:49.698895 31606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.1.keytab
--principal=kudu/127.12.12.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:44023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:49.700043 31606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:49.701175 31606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:49.711802 31614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:49.712044 31612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:49.712044 31611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:49.712954 31606 server_base.cc:1034] running on GCE node
I20260621 07:55:49.713505 31606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:49.714299 31606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:49.715616 31606 hybrid_clock.cc:648] HybridClock initialized: now 1782028549715542 us; error 71 us; skew 500 ppm
Jun 21 07:55:49 dist-test-slave-lbpt krb5kdc[31499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028549, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:49.720129 31606 init.cc:377] Logged in from keytab as kudu/127.12.12.1@KRBTEST.COM (short username kudu)
I20260621 07:55:49.721303 31606 webserver.cc:460] Webserver started at http://127.12.12.1:40301/ using document root <none> and password file <none>
I20260621 07:55:49.721992 31606 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:49.722093 31606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:49.722352 31606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:49.724718 31606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "4235936a1610434dab8b1f7d00f462d7"
format_stamp: "Formatted at 2026-06-21 07:55:49 on dist-test-slave-lbpt"
server_key: "7a03543ddafb51af2b6070e2b70f3338"
server_key_iv: "41573d207a6da55643483d59c1f260a0"
server_key_version: "encryptionkey@0"
I20260621 07:55:49.725379 31606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "4235936a1610434dab8b1f7d00f462d7"
format_stamp: "Formatted at 2026-06-21 07:55:49 on dist-test-slave-lbpt"
server_key: "7a03543ddafb51af2b6070e2b70f3338"
server_key_iv: "41573d207a6da55643483d59c1f260a0"
server_key_version: "encryptionkey@0"
I20260621 07:55:49.730252 31606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260621 07:55:49.733866 31621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:49.735743 31606 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:55:49.735950 31606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "4235936a1610434dab8b1f7d00f462d7"
format_stamp: "Formatted at 2026-06-21 07:55:49 on dist-test-slave-lbpt"
server_key: "7a03543ddafb51af2b6070e2b70f3338"
server_key_iv: "41573d207a6da55643483d59c1f260a0"
server_key_version: "encryptionkey@0"
I20260621 07:55:49.736114 31606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:49.785072 31606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:49.788784 31606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:49.789044 31606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:49.789733 31606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:49.790751 31606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:49.790825 31606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:49.790902 31606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:49.790953 31606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:49.802467 31606 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:43375
I20260621 07:55:49.802526 31734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:43375 every 8 connection(s)
I20260621 07:55:49.803568 31606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:55:49.813278 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 31606
I20260621 07:55:49.813410 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:55:49.813781 12336 external_mini_cluster.cc:1428] Setting key 50297e17f0d17b85014a5ac89d251912
Jun 21 07:55:49 dist-test-slave-lbpt krb5kdc[31499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028549, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.1@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:55:49.822036 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:49.805824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:34983 (local address 127.12.12.62:44023)
0621 07:55:49.806007 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:49.806012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:49.807411 (+ 1399us) server_negotiation.cc:407] Connection header received
0621 07:55:49.808667 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:49.808671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:49.808724 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:49.808803 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:49.810544 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:49.811116 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:49.812500 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:49.813033 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:49.813193 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:49.816511 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:49.816519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:49.816525 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:49.816564 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:49.818864 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:49.819844 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:49.819850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:49.819854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:49.819959 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:49.820524 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:49.820528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:49.820530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:49.820778 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:49.820866 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:49.821770 (+ 904us) server_negotiation.cc:299] Negotiation successful
0621 07:55:49.821904 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:55:49.823503 31737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:49.806306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:44023 (local address 127.12.12.1:34983)
0621 07:55:49.807193 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:49.807238 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:49.808286 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:49.809130 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:49.809138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:49.809690 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:49.810222 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:49.810237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:49.811309 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:49.811312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:49.812305 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:49.812316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:49.813260 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:49.813265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:49.813531 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:49.814257 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:55:49.814275 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:49.816304 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:49.819010 (+ 2706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:49.819020 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:49.819051 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:49.819641 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:49.820096 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:49.820103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:49.820108 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:49.820356 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:49.820907 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:49.820917 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:49.821422 (+ 505us) client_negotiation.cc:769] Sending connection context
0621 07:55:49.821696 (+ 274us) client_negotiation.cc:240] Negotiation successful
0621 07:55:49.822174 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":129,"threads_started":1}
I20260621 07:55:49.825266 31735 heartbeater.cc:347] Connected to a master server at 127.12.12.62:44023
WARNING: no policy specified for kudu/127.12.12.2@KRBTEST.COM; defaulting to no policy
I20260621 07:55:49.825560 31735 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:49.826131 31735 heartbeater.cc:511] Master 127.12.12.62:44023 requested a full tablet report, sending...
I20260621 07:55:49.828074 31544 ts_manager.cc:194] Registered new tserver with Master: 4235936a1610434dab8b1f7d00f462d7 (127.12.12.1:43375)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.2@KRBTEST.COM" created.
I20260621 07:55:49.829515 31544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.1@KRBTEST.COM'} at 127.12.12.1:34983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
Entry for principal kudu/127.12.12.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab.
I20260621 07:55:49.847116 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:44023
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:49.968982 31740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:49.969274 31740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:49.969321 31740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:49.973404 31740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:49.973464 31740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:49.973592 31740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:55:49.978956 31740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.2.keytab
--principal=kudu/127.12.12.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:44023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:49.980029 31740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:49.981015 31740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:49.991170 31745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:49.991225 31746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:49.991179 31748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:49.991585 31740 server_base.cc:1034] running on GCE node
I20260621 07:55:49.992136 31740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:49.993062 31740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:49.994257 31740 hybrid_clock.cc:648] HybridClock initialized: now 1782028549994235 us; error 44 us; skew 500 ppm
Jun 21 07:55:49 dist-test-slave-lbpt krb5kdc[31499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028549, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:49.998188 31740 init.cc:377] Logged in from keytab as kudu/127.12.12.2@KRBTEST.COM (short username kudu)
I20260621 07:55:49.999480 31740 webserver.cc:460] Webserver started at http://127.12.12.2:38015/ using document root <none> and password file <none>
I20260621 07:55:50.000335 31740 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:50.000442 31740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:50.000708 31740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:50.003959 31740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "b8b71dec05b3469a9e8b5e6ea2886314"
format_stamp: "Formatted at 2026-06-21 07:55:49 on dist-test-slave-lbpt"
server_key: "5d69664bcc9c43706838200dd6c3c024"
server_key_iv: "c67e0c1b78318e5edd6a42920ce58653"
server_key_version: "encryptionkey@0"
I20260621 07:55:50.004660 31740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "b8b71dec05b3469a9e8b5e6ea2886314"
format_stamp: "Formatted at 2026-06-21 07:55:49 on dist-test-slave-lbpt"
server_key: "5d69664bcc9c43706838200dd6c3c024"
server_key_iv: "c67e0c1b78318e5edd6a42920ce58653"
server_key_version: "encryptionkey@0"
I20260621 07:55:50.008991 31740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:55:50.011969 31755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:50.013741 31740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:55:50.013901 31740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "b8b71dec05b3469a9e8b5e6ea2886314"
format_stamp: "Formatted at 2026-06-21 07:55:49 on dist-test-slave-lbpt"
server_key: "5d69664bcc9c43706838200dd6c3c024"
server_key_iv: "c67e0c1b78318e5edd6a42920ce58653"
server_key_version: "encryptionkey@0"
I20260621 07:55:50.014035 31740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:50.048646 31740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:50.052386 31740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:50.052635 31740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:50.053305 31740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:50.054196 31740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:50.054270 31740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:50.054355 31740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:50.054402 31740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:50.066089 31740 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:33607
I20260621 07:55:50.066119 31868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:33607 every 8 connection(s)
I20260621 07:55:50.067137 31740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:55:50.075456 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 31740
I20260621 07:55:50.075580 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:55:50.075886 12336 external_mini_cluster.cc:1428] Setting key 77434c61e6b6695a42120a27fce9ea0e
Jun 21 07:55:50 dist-test-slave-lbpt krb5kdc[31499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028549, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.2@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:55:50.082751 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.069146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:42227 (local address 127.12.12.62:44023)
0621 07:55:50.069292 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:50.069295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:50.070320 (+ 1025us) server_negotiation.cc:407] Connection header received
0621 07:55:50.071373 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:50.071376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:50.071437 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:50.071548 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:50.072760 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.073298 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.074572 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.074794 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.074910 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.077974 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:50.077976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:50.077978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:50.078002 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:50.079852 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:50.080459 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:50.080465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:50.080470 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:50.080563 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:50.080885 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:50.080891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:50.080896 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:50.081161 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:50.081271 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:50.082305 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0621 07:55:50.082501 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:55:50.083702 31871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.069465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:44023 (local address 127.12.12.2:42227)
0621 07:55:50.070147 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:50.070188 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:50.071147 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:50.071740 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:50.071747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:50.072119 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:50.072601 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:50.072614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.073427 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.073430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:50.074374 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:50.074383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.074954 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.074957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:50.075164 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.075892 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:55:50.075910 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:50.077770 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:50.080009 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:50.080014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:50.080029 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:50.080352 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:50.080655 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:50.080657 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:50.080659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:50.080793 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:50.081414 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:50.081424 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:50.082007 (+ 583us) client_negotiation.cc:769] Sending connection context
0621 07:55:50.082215 (+ 208us) client_negotiation.cc:240] Negotiation successful
0621 07:55:50.082608 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":113,"threads_started":1}
I20260621 07:55:50.085137 31869 heartbeater.cc:347] Connected to a master server at 127.12.12.62:44023
I20260621 07:55:50.085381 31869 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:50.085947 31869 heartbeater.cc:511] Master 127.12.12.62:44023 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.12.3@KRBTEST.COM; defaulting to no policy
I20260621 07:55:50.087548 31544 ts_manager.cc:194] Registered new tserver with Master: b8b71dec05b3469a9e8b5e6ea2886314 (127.12.12.2:33607)
I20260621 07:55:50.088122 31544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.2@KRBTEST.COM'} at 127.12.12.2:42227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
Entry for principal kudu/127.12.12.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab.
I20260621 07:55:50.105767 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:44023
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:34291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:50.221601 31874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:50.222029 31874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:50.222117 31874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:50.228770 31874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:50.228886 31874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:50.228988 31874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:55:50.237366 31874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:34291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.3.keytab
--principal=kudu/127.12.12.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:44023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:50.238669 31874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:50.240129 31874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:50.252942 31880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:50.252935 31882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:50.253232 31874 server_base.cc:1034] running on GCE node
W20260621 07:55:50.253418 31879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:50.253878 31874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:50.254750 31874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:50.255942 31874 hybrid_clock.cc:648] HybridClock initialized: now 1782028550255914 us; error 46 us; skew 500 ppm
Jun 21 07:55:50 dist-test-slave-lbpt krb5kdc[31499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028550, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:50.259444 31874 init.cc:377] Logged in from keytab as kudu/127.12.12.3@KRBTEST.COM (short username kudu)
I20260621 07:55:50.260516 31874 webserver.cc:460] Webserver started at http://127.12.12.3:37751/ using document root <none> and password file <none>
I20260621 07:55:50.261184 31874 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:50.261248 31874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:50.261459 31874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:50.264166 31874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "9defcadca3584b08b7569ff97173b226"
format_stamp: "Formatted at 2026-06-21 07:55:50 on dist-test-slave-lbpt"
server_key: "b012ec51cfc31f8b14972a771e90d9a9"
server_key_iv: "1ce1619fb44e6ff39a749105ca32f12b"
server_key_version: "encryptionkey@0"
I20260621 07:55:50.264801 31874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "9defcadca3584b08b7569ff97173b226"
format_stamp: "Formatted at 2026-06-21 07:55:50 on dist-test-slave-lbpt"
server_key: "b012ec51cfc31f8b14972a771e90d9a9"
server_key_iv: "1ce1619fb44e6ff39a749105ca32f12b"
server_key_version: "encryptionkey@0"
I20260621 07:55:50.269593 31874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260621 07:55:50.272221 31889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:50.273550 31874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:55:50.273653 31874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "9defcadca3584b08b7569ff97173b226"
format_stamp: "Formatted at 2026-06-21 07:55:50 on dist-test-slave-lbpt"
server_key: "b012ec51cfc31f8b14972a771e90d9a9"
server_key_iv: "1ce1619fb44e6ff39a749105ca32f12b"
server_key_version: "encryptionkey@0"
I20260621 07:55:50.273785 31874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:50.287642 31874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:50.291360 31874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:50.291522 31874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:50.292099 31874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:50.293004 31874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:50.293063 31874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:50.293103 31874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:50.293118 31874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:50.303081 31874 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:43315
I20260621 07:55:50.303103 32002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:43315 every 8 connection(s)
I20260621 07:55:50.304052 31874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:55:50.312968 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 31874
I20260621 07:55:50.313086 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:55:50.313392 12336 external_mini_cluster.cc:1428] Setting key 9a38c67be5e935a13ebd005d34baf383
Jun 21 07:55:50 dist-test-slave-lbpt krb5kdc[31499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028550, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.3@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:55:50.319235 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.306082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:43639 (local address 127.12.12.62:44023)
0621 07:55:50.306260 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:50.306264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:50.307346 (+ 1082us) server_negotiation.cc:407] Connection header received
0621 07:55:50.308364 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:50.308368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:50.308433 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:50.308541 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:50.309788 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.310346 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.311689 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.311926 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.312046 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.314958 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:50.314963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:50.314966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:50.314999 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:50.316817 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:50.317352 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:50.317355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:50.317356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:50.317448 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:50.317949 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:50.317952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:50.317953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:50.318156 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:50.318244 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:50.318998 (+ 754us) server_negotiation.cc:299] Negotiation successful
0621 07:55:50.319108 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 07:55:50.320160 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.306485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:44023 (local address 127.12.12.3:43639)
0621 07:55:50.307177 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:50.307220 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:50.308110 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:50.308731 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:50.308738 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:50.309119 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:50.309629 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:50.309642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.310527 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.310529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:50.311551 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:50.311562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.312100 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.312104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:50.312280 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.312937 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:55:50.312953 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:50.314754 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:50.316938 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:50.316943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:50.316957 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:50.317238 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:50.317614 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:55:50.317622 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:55:50.317627 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:55:50.317806 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:55:50.318283 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:50.318291 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:50.318645 (+ 354us) client_negotiation.cc:769] Sending connection context
0621 07:55:50.318909 (+ 264us) client_negotiation.cc:240] Negotiation successful
0621 07:55:50.319197 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":127,"threads_started":1}
I20260621 07:55:50.321492 32003 heartbeater.cc:347] Connected to a master server at 127.12.12.62:44023
I20260621 07:55:50.321761 32003 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:50.322290 32003 heartbeater.cc:511] Master 127.12.12.62:44023 requested a full tablet report, sending...
I20260621 07:55:50.323523 31544 ts_manager.cc:194] Registered new tserver with Master: 9defcadca3584b08b7569ff97173b226 (127.12.12.3:43315)
I20260621 07:55:50.324195 31544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.3@KRBTEST.COM'} at 127.12.12.3:43639
I20260621 07:55:50.328001 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:55:50.337735 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.330203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.12.12.62:44023)
0621 07:55:50.330344 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:50.330349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:50.330362 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:55:50.330498 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:50.330503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:50.330556 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:50.330644 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:50.331369 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.331893 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.332754 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.332976 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.333083 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.334570 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:50.334574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:50.334576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:50.334599 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:50.335978 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:50.336460 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:50.336470 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:50.336472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:50.336546 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:50.336827 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:50.336830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:50.336832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:50.337061 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:50.337210 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:50.337446 (+ 236us) server_negotiation.cc:299] Negotiation successful
0621 07:55:50.337561 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:55:50.340706 31544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:55:50.343880 31544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:55:50.358749 32014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.352525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:33607 (local address 127.0.0.1:44556)
0621 07:55:50.352956 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:50.353006 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:50.353139 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:50.354585 (+ 1446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:50.354591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:50.354613 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:50.354715 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:50.354721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.355852 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.355858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:50.357024 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:50.357034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.358150 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.358153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:50.358259 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.358304 (+ 45us) client_negotiation.cc:769] Sending connection context
0621 07:55:50.358444 (+ 140us) client_negotiation.cc:240] Negotiation successful
0621 07:55:50.358571 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":5,"thread_start_us":97,"threads_started":1}
I20260621 07:55:50.358938 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.353420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.12.12.1:43375)
0621 07:55:50.353904 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:50.353912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:50.354144 (+ 232us) server_negotiation.cc:407] Connection header received
0621 07:55:50.354356 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:50.354362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:50.354584 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:50.354691 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:50.355195 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.356281 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.357665 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.358304 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.358454 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.358588 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:50.358685 (+ 97us) server_negotiation.cc:299] Negotiation successful
0621 07:55:50.358785 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":108,"threads_started":1}
I20260621 07:55:50.359050 32015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.353134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:43375 (local address 127.0.0.1:34314)
0621 07:55:50.354053 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:50.354069 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:50.354212 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:50.354966 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:50.354969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:50.354988 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:50.355068 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:50.355074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.356572 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.356575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:50.357554 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:50.357561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.358441 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.358444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:50.358550 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.358575 (+ 25us) client_negotiation.cc:769] Sending connection context
0621 07:55:50.358641 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 07:55:50.358702 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260621 07:55:50.359454 32017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.352536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:43315 (local address 127.0.0.1:33078)
0621 07:55:50.353437 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:50.353451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:50.353544 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:50.354333 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:50.354339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:50.354365 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:50.354555 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:50.354563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.355941 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.355944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:50.357110 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:50.357120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.358399 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.358403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:50.359153 (+ 750us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.359183 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 07:55:50.359264 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 07:55:50.359333 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":604,"spinlock_wait_cycles":6912,"thread_start_us":156,"threads_started":1}
I20260621 07:55:50.359535 32016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.352639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33078 (local address 127.12.12.3:43315)
0621 07:55:50.353477 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:50.353493 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:50.353521 (+ 28us) server_negotiation.cc:407] Connection header received
0621 07:55:50.353733 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:50.353744 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:50.353972 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:50.354131 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:50.354692 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.355778 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.357232 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.358015 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.358129 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.358281 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:50.359297 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0621 07:55:50.359395 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":94,"threads_started":1}
I20260621 07:55:50.358844 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.353338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44556 (local address 127.12.12.2:33607)
0621 07:55:50.354014 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:50.354026 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:50.354048 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:55:50.354136 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:50.354141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:50.354448 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:50.354603 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:50.354833 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.355719 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.357159 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.358015 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.358129 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.358276 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:50.358551 (+ 275us) server_negotiation.cc:299] Negotiation successful
0621 07:55:50.358667 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":131,"threads_started":1}
I20260621 07:55:50.362180 31937 tablet_service.cc:1467] Processing CreateTablet for tablet 875243594ee444479197bfd2d435ed09 (DEFAULT_TABLE table=test-table [id=824240f9eace4bcfa49cff8b3e086557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:50.362269 31669 tablet_service.cc:1467] Processing CreateTablet for tablet 875243594ee444479197bfd2d435ed09 (DEFAULT_TABLE table=test-table [id=824240f9eace4bcfa49cff8b3e086557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:50.362262 31803 tablet_service.cc:1467] Processing CreateTablet for tablet 875243594ee444479197bfd2d435ed09 (DEFAULT_TABLE table=test-table [id=824240f9eace4bcfa49cff8b3e086557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:50.363262 31937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875243594ee444479197bfd2d435ed09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:50.363288 31803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875243594ee444479197bfd2d435ed09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:50.363288 31669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875243594ee444479197bfd2d435ed09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:50.369706 32020 tablet_bootstrap.cc:492] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226: Bootstrap starting.
I20260621 07:55:50.371447 32022 tablet_bootstrap.cc:492] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314: Bootstrap starting.
I20260621 07:55:50.372490 32020 tablet_bootstrap.cc:654] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:50.373692 32022 tablet_bootstrap.cc:654] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:50.373790 32020 log.cc:824] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:50.374396 32021 tablet_bootstrap.cc:492] T 875243594ee444479197bfd2d435ed09 P 4235936a1610434dab8b1f7d00f462d7: Bootstrap starting.
I20260621 07:55:50.375017 32022 log.cc:824] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:50.376269 32020 tablet_bootstrap.cc:492] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226: No bootstrap required, opened a new log
I20260621 07:55:50.376473 32020 ts_tablet_manager.cc:1397] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260621 07:55:50.376796 32021 tablet_bootstrap.cc:654] T 875243594ee444479197bfd2d435ed09 P 4235936a1610434dab8b1f7d00f462d7: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:50.377166 32022 tablet_bootstrap.cc:492] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314: No bootstrap required, opened a new log
I20260621 07:55:50.377417 32022 ts_tablet_manager.cc:1397] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260621 07:55:50.377727 32021 log.cc:824] T 875243594ee444479197bfd2d435ed09 P 4235936a1610434dab8b1f7d00f462d7: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:50.379693 32021 tablet_bootstrap.cc:492] T 875243594ee444479197bfd2d435ed09 P 4235936a1610434dab8b1f7d00f462d7: No bootstrap required, opened a new log
I20260621 07:55:50.379930 32021 ts_tablet_manager.cc:1397] T 875243594ee444479197bfd2d435ed09 P 4235936a1610434dab8b1f7d00f462d7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260621 07:55:50.380647 32020 raft_consensus.cc:348] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9defcadca3584b08b7569ff97173b226" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 43315 } } peers { permanent_uuid: "b8b71dec05b3469a9e8b5e6ea2886314" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 33607 } } peers { permanent_uuid: "4235936a1610434dab8b1f7d00f462d7" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43375 } }
I20260621 07:55:50.380964 32020 raft_consensus.cc:374] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:50.381024 32020 raft_consensus.cc:729] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9defcadca3584b08b7569ff97173b226, State: Initialized, Role: FOLLOWER
I20260621 07:55:50.381280 32022 raft_consensus.cc:348] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9defcadca3584b08b7569ff97173b226" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 43315 } } peers { permanent_uuid: "b8b71dec05b3469a9e8b5e6ea2886314" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 33607 } } peers { permanent_uuid: "4235936a1610434dab8b1f7d00f462d7" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43375 } }
I20260621 07:55:50.381533 32022 raft_consensus.cc:374] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:50.381575 32022 raft_consensus.cc:729] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b71dec05b3469a9e8b5e6ea2886314, State: Initialized, Role: FOLLOWER
I20260621 07:55:50.381524 32020 consensus_queue.cc:260] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9defcadca3584b08b7569ff97173b226" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 43315 } } peers { permanent_uuid: "b8b71dec05b3469a9e8b5e6ea2886314" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 33607 } } peers { permanent_uuid: "4235936a1610434dab8b1f7d00f462d7" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43375 } }
I20260621 07:55:50.382014 32022 consensus_queue.cc:260] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9defcadca3584b08b7569ff97173b226" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 43315 } } peers { permanent_uuid: "b8b71dec05b3469a9e8b5e6ea2886314" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 33607 } } peers { permanent_uuid: "4235936a1610434dab8b1f7d00f462d7" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43375 } }
I20260621 07:55:50.382616 32020 ts_tablet_manager.cc:1428] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260621 07:55:50.382927 32022 ts_tablet_manager.cc:1428] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:55:50.382995 32003 heartbeater.cc:503] Master 127.12.12.62:44023 was elected leader, sending a full tablet report...
I20260621 07:55:50.383294 31869 heartbeater.cc:503] Master 127.12.12.62:44023 was elected leader, sending a full tablet report...
I20260621 07:55:50.384879 32021 raft_consensus.cc:348] T 875243594ee444479197bfd2d435ed09 P 4235936a1610434dab8b1f7d00f462d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9defcadca3584b08b7569ff97173b226" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 43315 } } peers { permanent_uuid: "b8b71dec05b3469a9e8b5e6ea2886314" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 33607 } } peers { permanent_uuid: "4235936a1610434dab8b1f7d00f462d7" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43375 } }
I20260621 07:55:50.385169 32021 raft_consensus.cc:374] T 875243594ee444479197bfd2d435ed09 P 4235936a1610434dab8b1f7d00f462d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:50.385258 32021 raft_consensus.cc:729] T 875243594ee444479197bfd2d435ed09 P 4235936a1610434dab8b1f7d00f462d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4235936a1610434dab8b1f7d00f462d7, State: Initialized, Role: FOLLOWER
I20260621 07:55:50.385736 32021 consensus_queue.cc:260] T 875243594ee444479197bfd2d435ed09 P 4235936a1610434dab8b1f7d00f462d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9defcadca3584b08b7569ff97173b226" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 43315 } } peers { permanent_uuid: "b8b71dec05b3469a9e8b5e6ea2886314" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 33607 } } peers { permanent_uuid: "4235936a1610434dab8b1f7d00f462d7" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43375 } }
I20260621 07:55:50.386456 31735 heartbeater.cc:503] Master 127.12.12.62:44023 was elected leader, sending a full tablet report...
I20260621 07:55:50.386842 32021 ts_tablet_manager.cc:1428] T 875243594ee444479197bfd2d435ed09 P 4235936a1610434dab8b1f7d00f462d7: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260621 07:55:50.480100 32027 raft_consensus.cc:482] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:55:50.480317 32027 raft_consensus.cc:504] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9defcadca3584b08b7569ff97173b226" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 43315 } } peers { permanent_uuid: "b8b71dec05b3469a9e8b5e6ea2886314" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 33607 } } peers { permanent_uuid: "4235936a1610434dab8b1f7d00f462d7" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43375 } }
I20260621 07:55:50.481714 32027 leader_election.cc:288] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9defcadca3584b08b7569ff97173b226 (127.12.12.3:43315), 4235936a1610434dab8b1f7d00f462d7 (127.12.12.1:43375)
I20260621 07:55:50.485698 31871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.481960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:43315 (local address 127.12.12.2:40627)
0621 07:55:50.482125 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:50.482143 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:50.482248 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:50.482605 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:50.482608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:50.482634 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:50.482759 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:50.482765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.483681 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.483686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:50.484702 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:50.484713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.485378 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.485380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:50.485459 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.485479 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 07:55:50.485539 (+ 60us) client_negotiation.cc:240] Negotiation successful
0621 07:55:50.485596 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260621 07:55:50.485822 32016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.482188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:40627 (local address 127.12.12.3:43315)
0621 07:55:50.482327 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:50.482332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:50.482348 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:55:50.482429 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:50.482434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:50.482487 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:50.482567 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:50.482876 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.483572 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.484831 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.485266 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.485370 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.485443 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:50.485585 (+ 142us) server_negotiation.cc:299] Negotiation successful
0621 07:55:50.485661 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:55:50.486393 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.482055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:43375 (local address 127.12.12.2:35759)
0621 07:55:50.482736 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:50.482750 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:50.482846 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:50.483127 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:50.483130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:50.483147 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:50.483266 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:50.483280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.484235 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.484238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:50.485176 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:50.485183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.486046 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.486050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:50.486151 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.486179 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 07:55:50.486239 (+ 60us) client_negotiation.cc:240] Negotiation successful
0621 07:55:50.486297 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":129,"threads_started":1}
I20260621 07:55:50.486497 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.482532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:35759 (local address 127.12.12.1:43375)
0621 07:55:50.482743 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:50.482750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:50.482807 (+ 57us) server_negotiation.cc:407] Connection header received
0621 07:55:50.482964 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:50.482967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:50.483017 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:50.483090 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:50.483410 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.484103 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.485298 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.485931 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.486040 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.486099 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:50.486276 (+ 177us) server_negotiation.cc:299] Negotiation successful
0621 07:55:50.486353 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260621 07:55:50.486544 31957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875243594ee444479197bfd2d435ed09" candidate_uuid: "b8b71dec05b3469a9e8b5e6ea2886314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9defcadca3584b08b7569ff97173b226" is_pre_election: true
I20260621 07:55:50.486891 31957 raft_consensus.cc:2454] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8b71dec05b3469a9e8b5e6ea2886314 in term 0.
I20260621 07:55:50.487324 31689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875243594ee444479197bfd2d435ed09" candidate_uuid: "b8b71dec05b3469a9e8b5e6ea2886314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4235936a1610434dab8b1f7d00f462d7" is_pre_election: true
I20260621 07:55:50.487588 31756 leader_election.cc:302] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9defcadca3584b08b7569ff97173b226, b8b71dec05b3469a9e8b5e6ea2886314; no voters:
I20260621 07:55:50.487658 31689 raft_consensus.cc:2454] T 875243594ee444479197bfd2d435ed09 P 4235936a1610434dab8b1f7d00f462d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8b71dec05b3469a9e8b5e6ea2886314 in term 0.
I20260621 07:55:50.487951 32027 raft_consensus.cc:2781] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:55:50.488056 32027 raft_consensus.cc:482] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:55:50.488117 32027 raft_consensus.cc:3037] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:50.489169 32027 raft_consensus.cc:504] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9defcadca3584b08b7569ff97173b226" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 43315 } } peers { permanent_uuid: "b8b71dec05b3469a9e8b5e6ea2886314" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 33607 } } peers { permanent_uuid: "4235936a1610434dab8b1f7d00f462d7" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43375 } }
I20260621 07:55:50.489573 32027 leader_election.cc:288] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [CANDIDATE]: Term 1 election: Requested vote from peers 9defcadca3584b08b7569ff97173b226 (127.12.12.3:43315), 4235936a1610434dab8b1f7d00f462d7 (127.12.12.1:43375)
I20260621 07:55:50.490023 31957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875243594ee444479197bfd2d435ed09" candidate_uuid: "b8b71dec05b3469a9e8b5e6ea2886314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9defcadca3584b08b7569ff97173b226"
I20260621 07:55:50.490167 31957 raft_consensus.cc:3037] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:50.490306 31689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875243594ee444479197bfd2d435ed09" candidate_uuid: "b8b71dec05b3469a9e8b5e6ea2886314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4235936a1610434dab8b1f7d00f462d7"
I20260621 07:55:50.490509 31689 raft_consensus.cc:3037] T 875243594ee444479197bfd2d435ed09 P 4235936a1610434dab8b1f7d00f462d7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:50.491425 31957 raft_consensus.cc:2454] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8b71dec05b3469a9e8b5e6ea2886314 in term 1.
I20260621 07:55:50.491670 31689 raft_consensus.cc:2454] T 875243594ee444479197bfd2d435ed09 P 4235936a1610434dab8b1f7d00f462d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8b71dec05b3469a9e8b5e6ea2886314 in term 1.
I20260621 07:55:50.491916 31756 leader_election.cc:302] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9defcadca3584b08b7569ff97173b226, b8b71dec05b3469a9e8b5e6ea2886314; no voters:
I20260621 07:55:50.492139 32027 raft_consensus.cc:2781] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:50.492365 32027 raft_consensus.cc:686] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [term 1 LEADER]: Becoming Leader. State: Replica: b8b71dec05b3469a9e8b5e6ea2886314, State: Running, Role: LEADER
I20260621 07:55:50.492743 32027 consensus_queue.cc:237] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9defcadca3584b08b7569ff97173b226" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 43315 } } peers { permanent_uuid: "b8b71dec05b3469a9e8b5e6ea2886314" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 33607 } } peers { permanent_uuid: "4235936a1610434dab8b1f7d00f462d7" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43375 } }
I20260621 07:55:50.496224 31543 catalog_manager.cc:5495] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8b71dec05b3469a9e8b5e6ea2886314 (127.12.12.2). New cstate: current_term: 1 leader_uuid: "b8b71dec05b3469a9e8b5e6ea2886314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9defcadca3584b08b7569ff97173b226" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 43315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8b71dec05b3469a9e8b5e6ea2886314" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 33607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4235936a1610434dab8b1f7d00f462d7" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43375 } health_report { overall_health: UNKNOWN } } }
I20260621 07:55:50.515025 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.510940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.12.12.2:33607)
0621 07:55:50.511105 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:50.511111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:50.511129 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:55:50.511231 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:50.511234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:50.511308 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:50.511459 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:55:50.511961 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.512477 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.513403 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.513620 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.513703 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.514029 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:55:50.514571 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:55:50.514683 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:50.514786 (+ 103us) server_negotiation.cc:299] Negotiation successful
0621 07:55:50.514842 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 07:55:50.521390 31689 raft_consensus.cc:1261] T 875243594ee444479197bfd2d435ed09 P 4235936a1610434dab8b1f7d00f462d7 [term 1 FOLLOWER]: Refusing update from remote peer b8b71dec05b3469a9e8b5e6ea2886314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:55:50.521390 31957 raft_consensus.cc:1261] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226 [term 1 FOLLOWER]: Refusing update from remote peer b8b71dec05b3469a9e8b5e6ea2886314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:55:50.522213 32027 consensus_queue.cc:1035] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4235936a1610434dab8b1f7d00f462d7" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 43375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:55:50.522468 32030 consensus_queue.cc:1035] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9defcadca3584b08b7569ff97173b226" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 43315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:55:50.531986 32033 mvcc.cc:205] Tried to move back new op lower bound from 7299188942927785984 to 7299188942821474304. Current Snapshot: MvccSnapshot[applied={T|T < 7299188942927785984}]
I20260621 07:55:50.541046 31542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:55:50.541257 31542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:55:50.544088 31542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9572ef67eb1248449b516f6884680c4b: Sending DeleteTablet for 3 replicas of tablet 875243594ee444479197bfd2d435ed09
I20260621 07:55:50.545082 31803 tablet_service.cc:1514] Processing DeleteTablet for tablet 875243594ee444479197bfd2d435ed09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:50 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:44556
I20260621 07:55:50.545081 31669 tablet_service.cc:1514] Processing DeleteTablet for tablet 875243594ee444479197bfd2d435ed09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:50 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:34314
I20260621 07:55:50.545089 31937 tablet_service.cc:1514] Processing DeleteTablet for tablet 875243594ee444479197bfd2d435ed09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:50 UTC) from {username='kudu', principal='kudu/127.12.12.62@KRBTEST.COM'} at 127.0.0.1:33078
I20260621 07:55:50.545768 32044 tablet_replica.cc:330] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226: stopping tablet replica
I20260621 07:55:50.545984 32042 tablet_replica.cc:330] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314: stopping tablet replica
I20260621 07:55:50.546133 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 31606
I20260621 07:55:50.546242 32044 raft_consensus.cc:2229] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:55:50.546340 32042 raft_consensus.cc:2229] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:55:50.546648 32044 raft_consensus.cc:2258] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:50.547047 32042 raft_consensus.cc:2258] T 875243594ee444479197bfd2d435ed09 P b8b71dec05b3469a9e8b5e6ea2886314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:50.548632 32044 ts_tablet_manager.cc:1905] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:55:50.552125 32044 ts_tablet_manager.cc:1918] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:55:50.552258 32044 log.cc:1196] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/875243594ee444479197bfd2d435ed09
I20260621 07:55:50.552618 32044 ts_tablet_manager.cc:1939] T 875243594ee444479197bfd2d435ed09 P 9defcadca3584b08b7569ff97173b226: Deleting consensus metadata
W20260621 07:55:50.553246 31529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.1:43375 (error 108)
I20260621 07:55:50.554102 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 31740
I20260621 07:55:50.555372 31528 catalog_manager.cc:4841] TS 9defcadca3584b08b7569ff97173b226 (127.12.12.3:43315): tablet 875243594ee444479197bfd2d435ed09 (table test-table [id=824240f9eace4bcfa49cff8b3e086557]) successfully deleted
I20260621 07:55:50.556414 32017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.556084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:43375 (local address 127.0.0.1:34320)
0621 07:55:50.556242 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:50.556322 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260621 07:55:50.556632 31529 catalog_manager.cc:4584] TS 4235936a1610434dab8b1f7d00f462d7 (127.12.12.1:43375): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 875243594ee444479197bfd2d435ed09: Network error: Client connection negotiation failed: client connection to 127.12.12.1:43375: connect: Connection refused (error 111)
I20260621 07:55:50.559962 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 31874
I20260621 07:55:50.561126 32017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.560184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:33607 (local address 127.0.0.1:44582)
0621 07:55:50.560944 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:50.561016 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.2:33607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":652}
I20260621 07:55:50.565795 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 31512
2026-06-21T07:55:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2454 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-21T07:55:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:55:50Z Disabled control of system clock
I20260621 07:55:50.597628 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:43139
--webserver_interface=127.12.12.62
--webserver_port=0
--builtin_ntp_servers=127.12.12.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:43139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:55:50.718590 32049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:50.718930 32049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:50.718971 32049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:50.723207 32049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:55:50.723292 32049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:50.723309 32049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:50.723326 32049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:55:50.723344 32049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:55:50.728577 32049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:43139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:43139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:50.729689 32049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:50.731150 32049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:55:50.740543 32049 server_base.cc:1034] running on GCE node
W20260621 07:55:50.740487 32057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:50.740504 32054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:50.740463 32055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:50.741438 32049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:50.742679 32049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:50.743922 32049 hybrid_clock.cc:648] HybridClock initialized: now 1782028550743887 us; error 47 us; skew 500 ppm
I20260621 07:55:50.745996 32049 webserver.cc:460] Webserver started at http://127.12.12.62:41845/ using document root <none> and password file <none>
I20260621 07:55:50.746659 32049 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:50.746728 32049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:50.746978 32049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:50.749107 32049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "0c0d20dd73bf47239ec346e3fafb429c"
format_stamp: "Formatted at 2026-06-21 07:55:50 on dist-test-slave-lbpt"
server_key: "45309c58bcf596a9ff0dcdff4b53ba0a"
server_key_iv: "22e69fef1c215673586ee510b58e9078"
server_key_version: "encryptionkey@0"
I20260621 07:55:50.749686 32049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "0c0d20dd73bf47239ec346e3fafb429c"
format_stamp: "Formatted at 2026-06-21 07:55:50 on dist-test-slave-lbpt"
server_key: "45309c58bcf596a9ff0dcdff4b53ba0a"
server_key_iv: "22e69fef1c215673586ee510b58e9078"
server_key_version: "encryptionkey@0"
I20260621 07:55:50.753554 32049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 07:55:50.756208 32063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:50.757515 32049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:55:50.757649 32049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "0c0d20dd73bf47239ec346e3fafb429c"
format_stamp: "Formatted at 2026-06-21 07:55:50 on dist-test-slave-lbpt"
server_key: "45309c58bcf596a9ff0dcdff4b53ba0a"
server_key_iv: "22e69fef1c215673586ee510b58e9078"
server_key_version: "encryptionkey@0"
I20260621 07:55:50.757835 32049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:50.776873 32049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:50.777644 32049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:50.777868 32049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:50.785801 32049 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:43139
I20260621 07:55:50.785805 32115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:43139 every 8 connection(s)
I20260621 07:55:50.786856 32049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:55:50.790118 32116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:50.794855 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 32049
I20260621 07:55:50.795022 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:55:50.795327 12336 external_mini_cluster.cc:1428] Setting key 6f1ab67296dfbc83d527e7d561799020
I20260621 07:55:50.796092 32116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c: Bootstrap starting.
I20260621 07:55:50.798585 32116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:50.799499 32116 log.cc:824] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:50.801677 32116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c: No bootstrap required, opened a new log
I20260621 07:55:50.805078 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:50.796832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42396 (local address 127.12.12.62:43139)
0621 07:55:50.797508 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:50.797523 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:50.797559 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:55:50.798253 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:50.798271 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:50.798799 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:50.799093 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:50.799448 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.800492 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.801648 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:50.802216 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:50.802344 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:50.803256 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:50.803261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:50.803273 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:50.803648 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:50.803702 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:50.803759 (+ 57us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:50.803838 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:50.803995 (+ 157us) server_negotiation.cc:299] Negotiation successful
0621 07:55:50.804290 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":160,"threads_started":1}
I20260621 07:55:50.805770 32116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0d20dd73bf47239ec346e3fafb429c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43139 } }
I20260621 07:55:50.805984 32116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:50.806041 32116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c0d20dd73bf47239ec346e3fafb429c, State: Initialized, Role: FOLLOWER
I20260621 07:55:50.806542 32116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0d20dd73bf47239ec346e3fafb429c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43139 } }
I20260621 07:55:50.806658 32116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:55:50.806697 32116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:55:50.806762 32116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:50.807703 32116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0d20dd73bf47239ec346e3fafb429c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43139 } }
I20260621 07:55:50.808039 32116 leader_election.cc:302] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c0d20dd73bf47239ec346e3fafb429c; no voters:
I20260621 07:55:50.808353 32116 leader_election.cc:288] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:55:50.808485 32121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:50.808694 32121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [term 1 LEADER]: Becoming Leader. State: Replica: 0c0d20dd73bf47239ec346e3fafb429c, State: Running, Role: LEADER
I20260621 07:55:50.809020 32121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0d20dd73bf47239ec346e3fafb429c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43139 } }
I20260621 07:55:50.809618 32116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:55:50.810662 32123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c0d20dd73bf47239ec346e3fafb429c. Latest consensus state: current_term: 1 leader_uuid: "0c0d20dd73bf47239ec346e3fafb429c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0d20dd73bf47239ec346e3fafb429c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43139 } } }
I20260621 07:55:50.810806 32123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:50.811085 32122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c0d20dd73bf47239ec346e3fafb429c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0d20dd73bf47239ec346e3fafb429c" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 43139 } } }
I20260621 07:55:50.811177 32122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:50.811287 32130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:55:50.814750 32130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:55:50.820257 32130 catalog_manager.cc:1347] Generated new cluster ID: 2bb319d64c7e420a8ed35ea943a9d09f
I20260621 07:55:50.820356 32130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:55:50.867002 32130 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:55:50.868008 32130 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:55:50.877178 32130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c: Generated new TSK 0
I20260621 07:55:50.877955 32130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:55:50.885010 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.1:0
--local_ip_for_outbound_sockets=127.12.12.1
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43139
--builtin_ntp_servers=127.12.12.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:55:51.003762 32140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:51.004067 32140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:51.004108 32140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:51.008329 32140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:51.008404 32140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:51.008462 32140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.1
I20260621 07:55:51.013793 32140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.1
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.1
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:51.014976 32140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:51.016404 32140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:51.026839 32145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:51.026787 32148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:51.026787 32146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:51.027127 32140 server_base.cc:1034] running on GCE node
I20260621 07:55:51.027756 32140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:51.028463 32140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:51.029667 32140 hybrid_clock.cc:648] HybridClock initialized: now 1782028551029635 us; error 52 us; skew 500 ppm
I20260621 07:55:51.035014 32140 webserver.cc:460] Webserver started at http://127.12.12.1:34065/ using document root <none> and password file <none>
I20260621 07:55:51.035665 32140 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:51.035777 32140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:51.036028 32140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:51.038189 32140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/data/instance:
uuid: "58445cd606ea46e5a5e460344e16ca27"
format_stamp: "Formatted at 2026-06-21 07:55:51 on dist-test-slave-lbpt"
server_key: "95df6ece702a12205acca7f7e4b7d5a1"
server_key_iv: "d1515c7b2cb741128b10e859fd452348"
server_key_version: "encryptionkey@0"
I20260621 07:55:51.038852 32140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance:
uuid: "58445cd606ea46e5a5e460344e16ca27"
format_stamp: "Formatted at 2026-06-21 07:55:51 on dist-test-slave-lbpt"
server_key: "95df6ece702a12205acca7f7e4b7d5a1"
server_key_iv: "d1515c7b2cb741128b10e859fd452348"
server_key_version: "encryptionkey@0"
I20260621 07:55:51.043058 32140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:55:51.045733 32154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:51.047421 32140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:55:51.047597 32140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/wal
uuid: "58445cd606ea46e5a5e460344e16ca27"
format_stamp: "Formatted at 2026-06-21 07:55:51 on dist-test-slave-lbpt"
server_key: "95df6ece702a12205acca7f7e4b7d5a1"
server_key_iv: "d1515c7b2cb741128b10e859fd452348"
server_key_version: "encryptionkey@0"
I20260621 07:55:51.047751 32140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:51.069904 32140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:51.070824 32140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:51.071036 32140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:51.071712 32140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:51.072695 32140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:51.072746 32140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:51.072788 32140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:51.072803 32140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:51.082067 32140 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.1:39785
I20260621 07:55:51.082101 32267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.1:39785 every 8 connection(s)
I20260621 07:55:51.083060 32140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/data/info.pb
I20260621 07:55:51.092326 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 32140
I20260621 07:55:51.092466 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-0/wal/instance
I20260621 07:55:51.092805 12336 external_mini_cluster.cc:1428] Setting key bff544e45a00380a70e68dddce9dff8b
I20260621 07:55:51.092814 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.084924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:53363 (local address 127.12.12.62:43139)
0621 07:55:51.085078 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:51.085083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:51.086225 (+ 1142us) server_negotiation.cc:407] Connection header received
0621 07:55:51.087706 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:51.087709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:51.087761 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:51.087897 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:51.088870 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.089440 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.090679 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.090964 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.091058 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.092204 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:51.092207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:51.092209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:51.092222 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:51.092243 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:51.092251 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:51.092330 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:51.092585 (+ 255us) server_negotiation.cc:299] Negotiation successful
0621 07:55:51.092656 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 07:55:51.093770 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.085405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43139 (local address 127.12.12.1:53363)
0621 07:55:51.086058 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:51.086113 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:51.087469 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:51.088073 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:51.088080 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:51.088187 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:51.088708 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:51.088720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.089548 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.089551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:51.090555 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:51.090565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.091112 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.091115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:51.091256 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.091885 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:55:51.091887 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:51.091927 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:55:51.091992 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:51.092359 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:51.092369 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:51.092375 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:55:51.092562 (+ 187us) client_negotiation.cc:240] Negotiation successful
0621 07:55:51.092906 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
I20260621 07:55:51.095049 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.2:0
--local_ip_for_outbound_sockets=127.12.12.2
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43139
--builtin_ntp_servers=127.12.12.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:55:51.095057 32268 heartbeater.cc:347] Connected to a master server at 127.12.12.62:43139
I20260621 07:55:51.095323 32268 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:51.095985 32268 heartbeater.cc:511] Master 127.12.12.62:43139 requested a full tablet report, sending...
I20260621 07:55:51.097743 32080 ts_manager.cc:194] Registered new tserver with Master: 58445cd606ea46e5a5e460344e16ca27 (127.12.12.1:39785)
I20260621 07:55:51.099512 32080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.12.1:53363
W20260621 07:55:51.212611 32271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:51.212945 32271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:51.213013 32271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:51.217644 32271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:51.217784 32271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:51.217888 32271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.2
I20260621 07:55:51.223259 32271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.2
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.2
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:51.224417 32271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:51.225874 32271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:51.236023 32276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:51.236028 32279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:51.236042 32277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:51.236580 32271 server_base.cc:1034] running on GCE node
I20260621 07:55:51.237165 32271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:51.237891 32271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:51.239077 32271 hybrid_clock.cc:648] HybridClock initialized: now 1782028551239056 us; error 35 us; skew 500 ppm
I20260621 07:55:51.241245 32271 webserver.cc:460] Webserver started at http://127.12.12.2:37933/ using document root <none> and password file <none>
I20260621 07:55:51.241914 32271 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:51.241983 32271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:51.242223 32271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:51.244527 32271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/data/instance:
uuid: "22add7c74c1845b59fed92f192b246bd"
format_stamp: "Formatted at 2026-06-21 07:55:51 on dist-test-slave-lbpt"
server_key: "a1fe35483c7fe26a58100f54ff6b0d18"
server_key_iv: "0642fcb2ec0e56c3e6aa686833225047"
server_key_version: "encryptionkey@0"
I20260621 07:55:51.245126 32271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance:
uuid: "22add7c74c1845b59fed92f192b246bd"
format_stamp: "Formatted at 2026-06-21 07:55:51 on dist-test-slave-lbpt"
server_key: "a1fe35483c7fe26a58100f54ff6b0d18"
server_key_iv: "0642fcb2ec0e56c3e6aa686833225047"
server_key_version: "encryptionkey@0"
I20260621 07:55:51.249086 32271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260621 07:55:51.252159 32285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:51.253669 32271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:55:51.253793 32271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/wal
uuid: "22add7c74c1845b59fed92f192b246bd"
format_stamp: "Formatted at 2026-06-21 07:55:51 on dist-test-slave-lbpt"
server_key: "a1fe35483c7fe26a58100f54ff6b0d18"
server_key_iv: "0642fcb2ec0e56c3e6aa686833225047"
server_key_version: "encryptionkey@0"
I20260621 07:55:51.254058 32271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:51.271075 32271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:51.271875 32271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:51.272091 32271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:51.272711 32271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:51.273725 32271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:51.273774 32271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:51.273813 32271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:51.273829 32271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:51.283463 32271 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.2:45111
I20260621 07:55:51.283486 32398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.2:45111 every 8 connection(s)
I20260621 07:55:51.284401 32271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/data/info.pb
I20260621 07:55:51.292868 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 32271
I20260621 07:55:51.293027 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/wal/instance
I20260621 07:55:51.293391 12336 external_mini_cluster.cc:1428] Setting key 8bd41f621655c840723a257ed5412732
I20260621 07:55:51.294776 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.286487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.2:56191 (local address 127.12.12.62:43139)
0621 07:55:51.286639 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:51.286643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:51.287812 (+ 1169us) server_negotiation.cc:407] Connection header received
0621 07:55:51.289352 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:51.289356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:51.289408 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:51.289489 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:51.290505 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.291151 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.292328 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.292551 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.292654 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.294177 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:51.294181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:51.294196 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:51.294210 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:51.294230 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:51.294239 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:51.294372 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:51.294584 (+ 212us) server_negotiation.cc:299] Negotiation successful
0621 07:55:51.294641 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:55:51.295670 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.3:0
--local_ip_for_outbound_sockets=127.12.12.3
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43139
--builtin_ntp_servers=127.12.12.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:55:51.295728 32401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.286919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43139 (local address 127.12.12.2:56191)
0621 07:55:51.287615 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:51.287659 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:51.289116 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:51.289653 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:51.289659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:51.289746 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:51.290279 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:51.290293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.291281 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.291284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:51.292183 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:51.292193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.292728 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.292734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:51.292964 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.293842 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:55:51.293844 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:51.293935 (+ 91us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:55:51.294003 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:51.294374 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:51.294382 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:51.294387 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 07:55:51.294562 (+ 175us) client_negotiation.cc:240] Negotiation successful
0621 07:55:51.294884 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":116,"threads_started":1}
I20260621 07:55:51.297044 32399 heartbeater.cc:347] Connected to a master server at 127.12.12.62:43139
I20260621 07:55:51.297329 32399 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:51.297907 32399 heartbeater.cc:511] Master 127.12.12.62:43139 requested a full tablet report, sending...
I20260621 07:55:51.298959 32080 ts_manager.cc:194] Registered new tserver with Master: 22add7c74c1845b59fed92f192b246bd (127.12.12.2:45111)
I20260621 07:55:51.299528 32080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.12.2:56191
W20260621 07:55:51.411383 32402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:51.411751 32402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:51.411828 32402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:51.416049 32402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:51.416142 32402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:51.416236 32402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.3
I20260621 07:55:51.421197 32402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.3
--webserver_port=0
--tserver_master_addrs=127.12.12.62:43139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.3
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:51.422465 32402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:51.423923 32402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:51.434084 32408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:51.434070 32407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:51.434070 32410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:51.434188 32402 server_base.cc:1034] running on GCE node
I20260621 07:55:51.434764 32402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:51.435506 32402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:51.436702 32402 hybrid_clock.cc:648] HybridClock initialized: now 1782028551436651 us; error 62 us; skew 500 ppm
I20260621 07:55:51.439143 32402 webserver.cc:460] Webserver started at http://127.12.12.3:40951/ using document root <none> and password file <none>
I20260621 07:55:51.439792 32402 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:51.439872 32402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:51.440042 32402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:51.442106 32402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/data/instance:
uuid: "651d067fa0cd4f7180b64759ffaf1ded"
format_stamp: "Formatted at 2026-06-21 07:55:51 on dist-test-slave-lbpt"
server_key: "ffd0b19ca727c0fd2d14f2daa71fd02f"
server_key_iv: "6e5ae7e191a82348a0e446738321f112"
server_key_version: "encryptionkey@0"
I20260621 07:55:51.442703 32402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance:
uuid: "651d067fa0cd4f7180b64759ffaf1ded"
format_stamp: "Formatted at 2026-06-21 07:55:51 on dist-test-slave-lbpt"
server_key: "ffd0b19ca727c0fd2d14f2daa71fd02f"
server_key_iv: "6e5ae7e191a82348a0e446738321f112"
server_key_version: "encryptionkey@0"
I20260621 07:55:51.446350 32402 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:55:51.448843 32416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:51.450160 32402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260621 07:55:51.450253 32402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/wal
uuid: "651d067fa0cd4f7180b64759ffaf1ded"
format_stamp: "Formatted at 2026-06-21 07:55:51 on dist-test-slave-lbpt"
server_key: "ffd0b19ca727c0fd2d14f2daa71fd02f"
server_key_iv: "6e5ae7e191a82348a0e446738321f112"
server_key_version: "encryptionkey@0"
I20260621 07:55:51.450366 32402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:51.468885 32402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:51.469609 32402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:51.469772 32402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:51.470316 32402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:55:51.471325 32402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:55:51.471370 32402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:51.471411 32402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:55:51.471427 32402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:51.480335 32402 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.3:45355
I20260621 07:55:51.480427 32529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.3:45355 every 8 connection(s)
I20260621 07:55:51.481307 32402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/data/info.pb
I20260621 07:55:51.482829 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 32402
I20260621 07:55:51.482963 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/wal/instance
I20260621 07:55:51.483278 12336 external_mini_cluster.cc:1428] Setting key d5fa9bb68d0dead7073ed8f08d35fa05
I20260621 07:55:51.491494 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.483288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.3:47267 (local address 127.12.12.62:43139)
0621 07:55:51.484741 (+ 1453us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:51.484745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:51.484759 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:55:51.486351 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:51.486353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:51.486399 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:51.486539 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:51.487464 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.488092 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.489347 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.489574 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.489666 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.490976 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:51.490980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:51.490982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:51.490995 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:51.491016 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:51.491024 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:51.491094 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:51.491293 (+ 199us) server_negotiation.cc:299] Negotiation successful
0621 07:55:51.491358 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1361}
I20260621 07:55:51.492412 32532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.483715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:43139 (local address 127.12.12.3:47267)
0621 07:55:51.484618 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:51.484675 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:51.486146 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:51.486708 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:51.486715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:51.486803 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:55:51.487329 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:51.487342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.488234 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.488237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:51.489197 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:51.489207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.489719 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.489722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:51.489921 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.490642 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:55:51.490643 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:55:51.490695 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:55:51.490766 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:55:51.491108 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:55:51.491123 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:55:51.491127 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 07:55:51.491234 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 07:55:51.491480 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":147,"threads_started":1}
I20260621 07:55:51.493726 32530 heartbeater.cc:347] Connected to a master server at 127.12.12.62:43139
I20260621 07:55:51.494001 32530 heartbeater.cc:464] Registering TS with master...
I20260621 07:55:51.494619 32530 heartbeater.cc:511] Master 127.12.12.62:43139 requested a full tablet report, sending...
I20260621 07:55:51.495723 32080 ts_manager.cc:194] Registered new tserver with Master: 651d067fa0cd4f7180b64759ffaf1ded (127.12.12.3:45355)
I20260621 07:55:51.496507 32080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.12.3:47267
I20260621 07:55:51.497671 12336 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:55:51.503886 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.499971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.12.12.62:43139)
0621 07:55:51.500092 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:51.500096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:51.500315 (+ 219us) server_negotiation.cc:407] Connection header received
0621 07:55:51.500427 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:51.500429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:51.500469 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:51.500541 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:51.500820 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.501345 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.502284 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.502571 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.502661 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.503432 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:51.503436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:51.503438 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:51.503448 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:51.503467 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:51.503475 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:51.503544 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:51.503597 (+ 53us) server_negotiation.cc:299] Negotiation successful
0621 07:55:51.503650 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:55:51.506649 32080 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:55:51.509728 32080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:55:51.524616 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.519113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:39785 (local address 127.0.0.1:33156)
0621 07:55:51.520328 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:51.520341 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:51.520440 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:51.520847 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:51.520850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:51.520867 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:51.520962 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:51.520969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.521914 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.521920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:51.523154 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:51.523163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.524061 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.524065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:51.524169 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.524208 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 07:55:51.524334 (+ 126us) client_negotiation.cc:240] Negotiation successful
0621 07:55:51.524453 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"mutex_wait_us":1,"thread_start_us":105,"threads_started":1}
I20260621 07:55:51.524650 32545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.519890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.12.12.1:39785)
0621 07:55:51.520240 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:51.520252 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:51.520450 (+ 198us) server_negotiation.cc:407] Connection header received
0621 07:55:51.520535 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:51.520540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:51.520709 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:51.520873 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:51.521059 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.521788 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.523279 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.523948 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.524051 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.524234 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:51.524361 (+ 127us) server_negotiation.cc:299] Negotiation successful
0621 07:55:51.524464 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":169,"threads_started":1}
I20260621 07:55:51.525478 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.518861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:45355 (local address 127.0.0.1:35656)
0621 07:55:51.519334 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:51.519373 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:51.519501 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:51.520946 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:51.520949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:51.520967 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:51.521045 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:51.521051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.522739 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.522742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:51.523824 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:51.523831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.525016 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.525020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:51.525107 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.525191 (+ 84us) client_negotiation.cc:769] Sending connection context
0621 07:55:51.525257 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 07:55:51.525325 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":47,"spinlock_wait_cycles":1920,"thread_start_us":103,"threads_started":1}
I20260621 07:55:51.525868 32546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.519895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.12.12.3:45355)
0621 07:55:51.520435 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:51.520446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:51.520470 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:55:51.520549 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:51.520554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:51.520742 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:51.521185 (+ 443us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:51.521415 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.522604 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.523944 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.524886 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.525189 (+ 303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.525380 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:51.525430 (+ 50us) server_negotiation.cc:299] Negotiation successful
0621 07:55:51.525708 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":172,"threads_started":1}
I20260621 07:55:51.526772 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.518645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:45111 (local address 127.0.0.1:45234)
0621 07:55:51.519048 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:51.519363 (+ 315us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:51.519502 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:51.520568 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:51.520574 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:51.520593 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:51.520735 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:51.520741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.522379 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.522383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:51.523594 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:51.523604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.524999 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.525003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:51.525104 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.525155 (+ 51us) client_negotiation.cc:769] Sending connection context
0621 07:55:51.525219 (+ 64us) client_negotiation.cc:240] Negotiation successful
0621 07:55:51.525285 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":5,"spinlock_wait_cycles":7296,"thread_start_us":100,"threads_started":1}
I20260621 07:55:51.527107 32544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.519350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.12.12.2:45111)
0621 07:55:51.519991 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:51.520005 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:51.520034 (+ 29us) server_negotiation.cc:407] Connection header received
0621 07:55:51.520114 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:51.520119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:51.520325 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:51.520518 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:51.520873 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.522242 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.523723 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.524473 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.524580 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.525401 (+ 821us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:51.525452 (+ 51us) server_negotiation.cc:299] Negotiation successful
0621 07:55:51.525546 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":121,"threads_started":1}
I20260621 07:55:51.527864 32202 tablet_service.cc:1467] Processing CreateTablet for tablet 76583c7505ae4588b4b98ad51b6df420 (DEFAULT_TABLE table=test-table [id=169485af715d4d3ebb04e6f4dae83b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:51.528290 32464 tablet_service.cc:1467] Processing CreateTablet for tablet 76583c7505ae4588b4b98ad51b6df420 (DEFAULT_TABLE table=test-table [id=169485af715d4d3ebb04e6f4dae83b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:51.528870 32202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76583c7505ae4588b4b98ad51b6df420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:51.529242 32464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76583c7505ae4588b4b98ad51b6df420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:51.533039 32333 tablet_service.cc:1467] Processing CreateTablet for tablet 76583c7505ae4588b4b98ad51b6df420 (DEFAULT_TABLE table=test-table [id=169485af715d4d3ebb04e6f4dae83b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:55:51.534107 32333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76583c7505ae4588b4b98ad51b6df420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:51.535638 32547 tablet_bootstrap.cc:492] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded: Bootstrap starting.
I20260621 07:55:51.537945 32547 tablet_bootstrap.cc:654] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:51.538784 32547 log.cc:824] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:51.539248 32548 tablet_bootstrap.cc:492] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27: Bootstrap starting.
I20260621 07:55:51.540977 32547 tablet_bootstrap.cc:492] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded: No bootstrap required, opened a new log
I20260621 07:55:51.541320 32547 ts_tablet_manager.cc:1397] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:55:51.541543 32548 tablet_bootstrap.cc:654] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:51.541833 32550 tablet_bootstrap.cc:492] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd: Bootstrap starting.
I20260621 07:55:51.542444 32548 log.cc:824] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:51.544004 32550 tablet_bootstrap.cc:654] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:51.544677 32548 tablet_bootstrap.cc:492] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27: No bootstrap required, opened a new log
I20260621 07:55:51.544806 32550 log.cc:824] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:51.544880 32548 ts_tablet_manager.cc:1397] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:55:51.545318 32547 raft_consensus.cc:348] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22add7c74c1845b59fed92f192b246bd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45111 } } peers { permanent_uuid: "651d067fa0cd4f7180b64759ffaf1ded" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45355 } } peers { permanent_uuid: "58445cd606ea46e5a5e460344e16ca27" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39785 } }
I20260621 07:55:51.545640 32547 raft_consensus.cc:374] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:51.545706 32547 raft_consensus.cc:729] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 651d067fa0cd4f7180b64759ffaf1ded, State: Initialized, Role: FOLLOWER
I20260621 07:55:51.546186 32547 consensus_queue.cc:260] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22add7c74c1845b59fed92f192b246bd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45111 } } peers { permanent_uuid: "651d067fa0cd4f7180b64759ffaf1ded" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45355 } } peers { permanent_uuid: "58445cd606ea46e5a5e460344e16ca27" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39785 } }
I20260621 07:55:51.546936 32530 heartbeater.cc:503] Master 127.12.12.62:43139 was elected leader, sending a full tablet report...
I20260621 07:55:51.547220 32550 tablet_bootstrap.cc:492] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd: No bootstrap required, opened a new log
I20260621 07:55:51.547230 32547 ts_tablet_manager.cc:1428] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260621 07:55:51.547405 32550 ts_tablet_manager.cc:1397] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:55:51.550561 32548 raft_consensus.cc:348] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22add7c74c1845b59fed92f192b246bd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45111 } } peers { permanent_uuid: "651d067fa0cd4f7180b64759ffaf1ded" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45355 } } peers { permanent_uuid: "58445cd606ea46e5a5e460344e16ca27" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39785 } }
I20260621 07:55:51.550868 32548 raft_consensus.cc:374] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:51.550940 32548 raft_consensus.cc:729] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58445cd606ea46e5a5e460344e16ca27, State: Initialized, Role: FOLLOWER
I20260621 07:55:51.551432 32548 consensus_queue.cc:260] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22add7c74c1845b59fed92f192b246bd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45111 } } peers { permanent_uuid: "651d067fa0cd4f7180b64759ffaf1ded" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45355 } } peers { permanent_uuid: "58445cd606ea46e5a5e460344e16ca27" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39785 } }
I20260621 07:55:51.551659 32550 raft_consensus.cc:348] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22add7c74c1845b59fed92f192b246bd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45111 } } peers { permanent_uuid: "651d067fa0cd4f7180b64759ffaf1ded" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45355 } } peers { permanent_uuid: "58445cd606ea46e5a5e460344e16ca27" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39785 } }
I20260621 07:55:51.551940 32550 raft_consensus.cc:374] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:51.552129 32268 heartbeater.cc:503] Master 127.12.12.62:43139 was elected leader, sending a full tablet report...
I20260621 07:55:51.552474 32548 ts_tablet_manager.cc:1428] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260621 07:55:51.552572 32550 raft_consensus.cc:729] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22add7c74c1845b59fed92f192b246bd, State: Initialized, Role: FOLLOWER
I20260621 07:55:51.553025 32550 consensus_queue.cc:260] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22add7c74c1845b59fed92f192b246bd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45111 } } peers { permanent_uuid: "651d067fa0cd4f7180b64759ffaf1ded" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45355 } } peers { permanent_uuid: "58445cd606ea46e5a5e460344e16ca27" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39785 } }
I20260621 07:55:51.553717 32399 heartbeater.cc:503] Master 127.12.12.62:43139 was elected leader, sending a full tablet report...
I20260621 07:55:51.553987 32550 ts_tablet_manager.cc:1428] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
W20260621 07:55:51.584352 32269 tablet.cc:2369] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:55:51.725297 32554 raft_consensus.cc:482] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:55:51.725533 32554 raft_consensus.cc:504] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22add7c74c1845b59fed92f192b246bd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45111 } } peers { permanent_uuid: "651d067fa0cd4f7180b64759ffaf1ded" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45355 } } peers { permanent_uuid: "58445cd606ea46e5a5e460344e16ca27" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39785 } }
I20260621 07:55:51.726913 32554 leader_election.cc:288] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22add7c74c1845b59fed92f192b246bd (127.12.12.2:45111), 651d067fa0cd4f7180b64759ffaf1ded (127.12.12.3:45355)
W20260621 07:55:51.732203 32531 tablet.cc:2369] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:55:51.732474 32546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.727332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:41711 (local address 127.12.12.3:45355)
0621 07:55:51.727483 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:51.727489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:51.727760 (+ 271us) server_negotiation.cc:407] Connection header received
0621 07:55:51.727854 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:51.727856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:51.727928 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:51.728036 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:51.728450 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.729266 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.731244 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.731853 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.731958 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.732022 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:51.732252 (+ 230us) server_negotiation.cc:299] Negotiation successful
0621 07:55:51.732355 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 07:55:51.732477 32557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.727193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.3:45355 (local address 127.12.12.1:41711)
0621 07:55:51.727599 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:51.727617 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:51.727752 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:51.728046 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:51.728049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:51.728087 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:51.728302 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:51.728308 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.729386 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.729390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:51.731100 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:51.731112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.731996 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.732000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:51.732113 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.732140 (+ 27us) client_negotiation.cc:769] Sending connection context
0621 07:55:51.732224 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 07:55:51.732318 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":260,"spinlock_wait_cycles":10624,"thread_start_us":129,"threads_started":1}
I20260621 07:55:51.733158 32482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76583c7505ae4588b4b98ad51b6df420" candidate_uuid: "58445cd606ea46e5a5e460344e16ca27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "651d067fa0cd4f7180b64759ffaf1ded" is_pre_election: true
I20260621 07:55:51.733311 32556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.727193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.2:45111 (local address 127.12.12.1:33325)
0621 07:55:51.727691 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:51.727703 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:55:51.727828 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:55:51.728111 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:55:51.728113 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:55:51.728148 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:55:51.728289 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:51.728297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.729386 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.729390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:51.731100 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:55:51.731112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.731996 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.732000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:55:51.732988 (+ 988us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.733012 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 07:55:51.733092 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 07:55:51.733164 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":1016,"spinlock_wait_cycles":3712,"thread_start_us":91,"threads_started":1}
I20260621 07:55:51.733292 32544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.727320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.12.1:33325 (local address 127.12.12.2:45111)
0621 07:55:51.727483 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:51.727489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:51.727776 (+ 287us) server_negotiation.cc:407] Connection header received
0621 07:55:51.727939 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:51.727942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:51.728008 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:51.728090 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:55:51.728444 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.729250 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.731245 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.731845 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.731958 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.732022 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:51.733124 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0621 07:55:51.733195 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 07:55:51.733518 32482 raft_consensus.cc:2454] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58445cd606ea46e5a5e460344e16ca27 in term 0.
I20260621 07:55:51.733922 32353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76583c7505ae4588b4b98ad51b6df420" candidate_uuid: "58445cd606ea46e5a5e460344e16ca27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22add7c74c1845b59fed92f192b246bd" is_pre_election: true
I20260621 07:55:51.734067 32158 leader_election.cc:302] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58445cd606ea46e5a5e460344e16ca27, 651d067fa0cd4f7180b64759ffaf1ded; no voters:
I20260621 07:55:51.734189 32353 raft_consensus.cc:2454] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58445cd606ea46e5a5e460344e16ca27 in term 0.
I20260621 07:55:51.734453 32554 raft_consensus.cc:2781] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:55:51.734580 32554 raft_consensus.cc:482] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:55:51.734638 32554 raft_consensus.cc:3037] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:51.735703 32554 raft_consensus.cc:504] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22add7c74c1845b59fed92f192b246bd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45111 } } peers { permanent_uuid: "651d067fa0cd4f7180b64759ffaf1ded" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45355 } } peers { permanent_uuid: "58445cd606ea46e5a5e460344e16ca27" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39785 } }
I20260621 07:55:51.736068 32554 leader_election.cc:288] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [CANDIDATE]: Term 1 election: Requested vote from peers 22add7c74c1845b59fed92f192b246bd (127.12.12.2:45111), 651d067fa0cd4f7180b64759ffaf1ded (127.12.12.3:45355)
I20260621 07:55:51.736451 32353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76583c7505ae4588b4b98ad51b6df420" candidate_uuid: "58445cd606ea46e5a5e460344e16ca27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22add7c74c1845b59fed92f192b246bd"
I20260621 07:55:51.736585 32353 raft_consensus.cc:3037] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:51.736626 32482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76583c7505ae4588b4b98ad51b6df420" candidate_uuid: "58445cd606ea46e5a5e460344e16ca27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "651d067fa0cd4f7180b64759ffaf1ded"
I20260621 07:55:51.736785 32482 raft_consensus.cc:3037] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:51.737900 32482 raft_consensus.cc:2454] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58445cd606ea46e5a5e460344e16ca27 in term 1.
I20260621 07:55:51.738242 32158 leader_election.cc:302] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58445cd606ea46e5a5e460344e16ca27, 651d067fa0cd4f7180b64759ffaf1ded; no voters:
I20260621 07:55:51.738283 32353 raft_consensus.cc:2454] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58445cd606ea46e5a5e460344e16ca27 in term 1.
I20260621 07:55:51.738585 32554 raft_consensus.cc:2781] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:51.738888 32554 raft_consensus.cc:686] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [term 1 LEADER]: Becoming Leader. State: Replica: 58445cd606ea46e5a5e460344e16ca27, State: Running, Role: LEADER
I20260621 07:55:51.739264 32554 consensus_queue.cc:237] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22add7c74c1845b59fed92f192b246bd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45111 } } peers { permanent_uuid: "651d067fa0cd4f7180b64759ffaf1ded" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45355 } } peers { permanent_uuid: "58445cd606ea46e5a5e460344e16ca27" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39785 } }
I20260621 07:55:51.742893 32078 catalog_manager.cc:5495] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58445cd606ea46e5a5e460344e16ca27 (127.12.12.1). New cstate: current_term: 1 leader_uuid: "58445cd606ea46e5a5e460344e16ca27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22add7c74c1845b59fed92f192b246bd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "651d067fa0cd4f7180b64759ffaf1ded" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58445cd606ea46e5a5e460344e16ca27" member_type: VOTER last_known_addr { host: "127.12.12.1" port: 39785 } health_report { overall_health: HEALTHY } } }
I20260621 07:55:51.763458 32545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.759614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33172 (local address 127.12.12.1:39785)
0621 07:55:51.759753 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:51.759757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:51.759770 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:55:51.759900 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:51.759903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:51.759962 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:51.760061 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:55:51.760311 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.760807 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.761826 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:51.762041 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:51.762128 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:51.762588 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:55:51.763063 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:55:51.763171 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:51.763259 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 07:55:51.763309 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:55:51.769693 32482 raft_consensus.cc:1261] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded [term 1 FOLLOWER]: Refusing update from remote peer 58445cd606ea46e5a5e460344e16ca27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:55:51.769693 32353 raft_consensus.cc:1261] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd [term 1 FOLLOWER]: Refusing update from remote peer 58445cd606ea46e5a5e460344e16ca27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:55:51.770519 32554 consensus_queue.cc:1035] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22add7c74c1845b59fed92f192b246bd" member_type: VOTER last_known_addr { host: "127.12.12.2" port: 45111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:55:51.770828 32558 consensus_queue.cc:1035] T 76583c7505ae4588b4b98ad51b6df420 P 58445cd606ea46e5a5e460344e16ca27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "651d067fa0cd4f7180b64759ffaf1ded" member_type: VOTER last_known_addr { host: "127.12.12.3" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:55:51.778860 32562 mvcc.cc:205] Tried to move back new op lower bound from 7299188948039868416 to 7299188947926925312. Current Snapshot: MvccSnapshot[applied={T|T < 7299188948039868416}]
I20260621 07:55:51.779484 32563 mvcc.cc:205] Tried to move back new op lower bound from 7299188948039868416 to 7299188947926925312. Current Snapshot: MvccSnapshot[applied={T|T < 7299188948039868416}]
I20260621 07:55:51.788470 32079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:55:51.788655 32079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:55:51.791426 32079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c0d20dd73bf47239ec346e3fafb429c: Sending DeleteTablet for 3 replicas of tablet 76583c7505ae4588b4b98ad51b6df420
I20260621 07:55:51.792284 32333 tablet_service.cc:1514] Processing DeleteTablet for tablet 76583c7505ae4588b4b98ad51b6df420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:51 UTC) from {username='slave'} at 127.0.0.1:45234
I20260621 07:55:51.792434 32202 tablet_service.cc:1514] Processing DeleteTablet for tablet 76583c7505ae4588b4b98ad51b6df420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:51 UTC) from {username='slave'} at 127.0.0.1:33156
I20260621 07:55:51.793537 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 32140
I20260621 07:55:51.793514 32464 tablet_service.cc:1514] Processing DeleteTablet for tablet 76583c7505ae4588b4b98ad51b6df420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:55:51 UTC) from {username='slave'} at 127.0.0.1:35656
I20260621 07:55:51.795614 32571 tablet_replica.cc:330] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd: stopping tablet replica
I20260621 07:55:51.795953 32571 raft_consensus.cc:2229] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:55:51.796291 32571 raft_consensus.cc:2258] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:51.796710 32573 tablet_replica.cc:330] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded: stopping tablet replica
I20260621 07:55:51.796993 32573 raft_consensus.cc:2229] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:55:51.797278 32573 raft_consensus.cc:2258] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:55:51.798015 32571 ts_tablet_manager.cc:1905] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:55:51.799716 32573 ts_tablet_manager.cc:1905] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:55:51.800742 32571 ts_tablet_manager.cc:1918] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:55:51.800843 32571 log.cc:1196] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-1/wal/wals/76583c7505ae4588b4b98ad51b6df420
I20260621 07:55:51.801139 32571 ts_tablet_manager.cc:1939] T 76583c7505ae4588b4b98ad51b6df420 P 22add7c74c1845b59fed92f192b246bd: Deleting consensus metadata
I20260621 07:55:51.802496 32064 catalog_manager.cc:4841] TS 22add7c74c1845b59fed92f192b246bd (127.12.12.2:45111): tablet 76583c7505ae4588b4b98ad51b6df420 (table test-table [id=169485af715d4d3ebb04e6f4dae83b74]) successfully deleted
W20260621 07:55:51.803308 32066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.1:39785 (error 108)
I20260621 07:55:51.803500 32573 ts_tablet_manager.cc:1918] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:55:51.803607 32573 log.cc:1196] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded: Deleting WAL directory at /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028299065733-12336-0/minicluster-data/ts-2/wal/wals/76583c7505ae4588b4b98ad51b6df420
I20260621 07:55:51.803925 32573 ts_tablet_manager.cc:1939] T 76583c7505ae4588b4b98ad51b6df420 P 651d067fa0cd4f7180b64759ffaf1ded: Deleting consensus metadata
I20260621 07:55:51.804260 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 32271
I20260621 07:55:51.805234 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:51.804548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.1:39785 (local address 127.0.0.1:33178)
0621 07:55:51.805074 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0621 07:55:51.805150 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.1:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431}
I20260621 07:55:51.805627 32067 catalog_manager.cc:4841] TS 651d067fa0cd4f7180b64759ffaf1ded (127.12.12.3:45355): tablet 76583c7505ae4588b4b98ad51b6df420 (table test-table [id=169485af715d4d3ebb04e6f4dae83b74]) successfully deleted
W20260621 07:55:51.805938 32066 catalog_manager.cc:4584] TS 58445cd606ea46e5a5e460344e16ca27 (127.12.12.1:39785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76583c7505ae4588b4b98ad51b6df420: Network error: Client connection negotiation failed: client connection to 127.12.12.1:39785: connect: Connection refused (error 111)
I20260621 07:55:51.810526 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 32402
I20260621 07:55:51.817147 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 32049
2026-06-21T07:55:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1250 ms)
[----------] 6 tests from EncryptionPolicyTest (10729 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:55:51 dist-test-slave-lbpt krb5kdc[32577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:55:51 dist-test-slave-lbpt krb5kdc[32577](info): set up 2 sockets
Jun 21 07:55:51 dist-test-slave-lbpt krb5kdc[32577](info): commencing operation
krb5kdc: starting...
W20260621 07:55:52.870723 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:55:52 dist-test-slave-lbpt krb5kdc[32577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:55:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:55:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:55:53.007551 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:53.120055 32590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:53.120393 32590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:53.120466 32590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:53.124640 32590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:55:53.124763 32590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:53.124802 32590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:53.124841 32590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:55:53.124876 32590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:55:53.130903 32590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37607
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:53.132128 32590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:53.133239 32590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:55:53.142330 32590 server_base.cc:1034] running on GCE node
W20260621 07:55:53.142256 32595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:53.142248 32596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:53.142248 32598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:53.143189 32590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:53.144445 32590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:53.145623 32590 hybrid_clock.cc:648] HybridClock initialized: now 1782028553145605 us; error 37 us; skew 500 ppm
Jun 21 07:55:53 dist-test-slave-lbpt krb5kdc[32577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:53.148736 32590 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:55:53.149714 32590 webserver.cc:460] Webserver started at http://127.0.0.1:35671/ using document root <none> and password file <none>
I20260621 07:55:53.150337 32590 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:53.150454 32590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:53.150736 32590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:53.152884 32590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "31cc5dee07614377a87c1d322f45bb13"
format_stamp: "Formatted at 2026-06-21 07:55:53 on dist-test-slave-lbpt"
server_key: "c7888dd21c096369485bc084abe08105"
server_key_iv: "a3356ff89888612d862323fd469f2253"
server_key_version: "encryptionkey@0"
I20260621 07:55:53.153465 32590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "31cc5dee07614377a87c1d322f45bb13"
format_stamp: "Formatted at 2026-06-21 07:55:53 on dist-test-slave-lbpt"
server_key: "c7888dd21c096369485bc084abe08105"
server_key_iv: "a3356ff89888612d862323fd469f2253"
server_key_version: "encryptionkey@0"
I20260621 07:55:53.157497 32590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260621 07:55:53.160225 32605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:53.161617 32590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:55:53.161799 32590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "31cc5dee07614377a87c1d322f45bb13"
format_stamp: "Formatted at 2026-06-21 07:55:53 on dist-test-slave-lbpt"
server_key: "c7888dd21c096369485bc084abe08105"
server_key_iv: "a3356ff89888612d862323fd469f2253"
server_key_version: "encryptionkey@0"
I20260621 07:55:53.161937 32590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:53.174705 32590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:53.178339 32590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:53.178599 32590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:53.186507 32590 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37607
I20260621 07:55:53.186542 32657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37607 every 8 connection(s)
I20260621 07:55:53.187546 32590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:55:53.190591 32658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:53.194741 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 32590
I20260621 07:55:53.194893 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:55:53.195180 12336 external_mini_cluster.cc:1428] Setting key eda2a7f8362349436271eaae81caab2f
I20260621 07:55:53.197278 32658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13: Bootstrap starting.
I20260621 07:55:53.199925 32658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:53.200700 32658 log.cc:824] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:53.202831 32658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13: No bootstrap required, opened a new log
Jun 21 07:55:53 dist-test-slave-lbpt krb5kdc[32577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 07:55:53.207623 32658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cc5dee07614377a87c1d322f45bb13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37607 } }
I20260621 07:55:53.207965 32658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:53.208020 32658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31cc5dee07614377a87c1d322f45bb13, State: Initialized, Role: FOLLOWER
I20260621 07:55:53.208592 32658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cc5dee07614377a87c1d322f45bb13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37607 } }
I20260621 07:55:53.208776 32658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:55:53.208873 32658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:55:53.208998 32658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:53.210299 32658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cc5dee07614377a87c1d322f45bb13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37607 } }
I20260621 07:55:53.210762 32658 leader_election.cc:302] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31cc5dee07614377a87c1d322f45bb13; no voters:
I20260621 07:55:53.211182 32658 leader_election.cc:288] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:55:53.211431 32663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:53.211687 32663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [term 1 LEADER]: Becoming Leader. State: Replica: 31cc5dee07614377a87c1d322f45bb13, State: Running, Role: LEADER
I20260621 07:55:53.212035 32663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cc5dee07614377a87c1d322f45bb13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37607 } }
I20260621 07:55:53.212425 32661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:53.196827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51267 (local address 127.0.0.1:37607)
0621 07:55:53.197271 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:53.197288 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:53.197326 (+ 38us) server_negotiation.cc:407] Connection header received
0621 07:55:53.198065 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:53.198085 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:53.198515 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:53.198828 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:53.199797 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:53.201000 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:53.201909 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:53.202488 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:53.202661 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:53.206130 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:53.206139 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:53.206160 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:53.206197 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:53.208434 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:53.209233 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:53.209242 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:53.209251 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:53.209416 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:53.209801 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:53.209804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:53.209806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:53.210451 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:53.210729 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:53.211065 (+ 336us) server_negotiation.cc:299] Negotiation successful
0621 07:55:53.211397 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":154,"threads_started":1}
I20260621 07:55:53.212644 32658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:55:53.213325 32663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31cc5dee07614377a87c1d322f45bb13. Latest consensus state: current_term: 1 leader_uuid: "31cc5dee07614377a87c1d322f45bb13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cc5dee07614377a87c1d322f45bb13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37607 } } }
I20260621 07:55:53.213449 32663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:53.213742 32665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31cc5dee07614377a87c1d322f45bb13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cc5dee07614377a87c1d322f45bb13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37607 } } }
I20260621 07:55:53.213847 32665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13 [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:53.214095 32671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:55:53.216207 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:55:53.220122 32671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:55:53.226684 32671 catalog_manager.cc:1347] Generated new cluster ID: 051d3d8df54a43bb992ec74cb640681b
I20260621 07:55:53.226768 32671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:55:53.228245 32661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:53.218594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48035 (local address 127.0.0.1:37607)
0621 07:55:53.218723 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:53.218728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:53.218909 (+ 181us) server_negotiation.cc:407] Connection header received
0621 07:55:53.219049 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:53.219052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:53.219102 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:53.219196 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:53.220039 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:53.220869 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:53.222213 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:53.222585 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:53.222648 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:53.224237 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:53.224241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:53.224244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:53.224269 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:53.226237 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:53.226861 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:53.226865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:53.226867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:53.226952 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:53.227279 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:53.227282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:53.227284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:53.227479 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:53.227668 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:53.227937 (+ 269us) server_negotiation.cc:299] Negotiation successful
0621 07:55:53.228050 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":45}
I20260621 07:55:53.252396 32671 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:55:53.253681 32671 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:55:53.262575 32671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31cc5dee07614377a87c1d322f45bb13: Generated new TSK 0
I20260621 07:55:53.263441 32671 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:55:53.273379 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 32590
W20260621 07:55:53.279003 32575 connection.cc:535] client connection to 127.0.0.1:37607 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37607): Connection reset by peer (error 104)
2026-06-21T07:55:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:55:53 dist-test-slave-lbpt krb5kdc[32689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:55:53 dist-test-slave-lbpt krb5kdc[32689](info): set up 2 sockets
Jun 21 07:55:53 dist-test-slave-lbpt krb5kdc[32689](info): commencing operation
krb5kdc: starting...
W20260621 07:55:54.326751 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:55:54 dist-test-slave-lbpt krb5kdc[32689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:55:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:55:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:55:54.455587 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:54.567842 32702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:54.568131 32702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:54.568176 32702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:54.572364 32702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:55:54.572441 32702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:54.572463 32702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:54.572479 32702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:55:54.572495 32702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:55:54.578116 32702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:54.579366 32702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:54.580473 32702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:54.589255 32707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:54.589291 32708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:54.589323 32702 server_base.cc:1034] running on GCE node
W20260621 07:55:54.589284 32710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:54.590204 32702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:54.591420 32702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:54.592612 32702 hybrid_clock.cc:648] HybridClock initialized: now 1782028554592594 us; error 30 us; skew 500 ppm
Jun 21 07:55:54 dist-test-slave-lbpt krb5kdc[32689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:54.595480 32702 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:55:54.596458 32702 webserver.cc:460] Webserver started at http://127.0.0.1:39753/ using document root <none> and password file <none>
I20260621 07:55:54.597033 32702 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:54.597126 32702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:54.597366 32702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:54.599567 32702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "59f26b46822e438abb594c541bff5be2"
format_stamp: "Formatted at 2026-06-21 07:55:54 on dist-test-slave-lbpt"
server_key: "350b0cc7c0b02a8cc335904a7c3a08f3"
server_key_iv: "fe3d2b6b3d40caa3f92bf8f7ad767d80"
server_key_version: "encryptionkey@0"
I20260621 07:55:54.600196 32702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "59f26b46822e438abb594c541bff5be2"
format_stamp: "Formatted at 2026-06-21 07:55:54 on dist-test-slave-lbpt"
server_key: "350b0cc7c0b02a8cc335904a7c3a08f3"
server_key_iv: "fe3d2b6b3d40caa3f92bf8f7ad767d80"
server_key_version: "encryptionkey@0"
I20260621 07:55:54.604434 32702 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260621 07:55:54.607249 32717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:54.608667 32702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:55:54.608855 32702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "59f26b46822e438abb594c541bff5be2"
format_stamp: "Formatted at 2026-06-21 07:55:54 on dist-test-slave-lbpt"
server_key: "350b0cc7c0b02a8cc335904a7c3a08f3"
server_key_iv: "fe3d2b6b3d40caa3f92bf8f7ad767d80"
server_key_version: "encryptionkey@0"
I20260621 07:55:54.608987 32702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:54.634222 32702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:54.637545 32702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:54.637797 32702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:54.645856 32702 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39973
I20260621 07:55:54.645874 301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39973 every 8 connection(s)
I20260621 07:55:54.646946 32702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:55:54.649843 302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:54.652838 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 32702
I20260621 07:55:54.653004 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:55:54.653360 12336 external_mini_cluster.cc:1428] Setting key 1f2126edea9a00a6e91fba60561022d9
I20260621 07:55:54.656064 302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2: Bootstrap starting.
I20260621 07:55:54.658607 302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:54.659765 302 log.cc:824] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:54.662119 302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2: No bootstrap required, opened a new log
I20260621 07:55:54.663153 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:54.654978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54715 (local address 127.0.0.1:39973)
0621 07:55:54.655408 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:54.655424 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:54.655460 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:55:54.656111 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:54.656157 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:54.656545 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:54.656826 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:54.657210 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:54.658237 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:54.659671 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:54.660309 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:54.660489 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:54.661378 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:54.661383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:54.661396 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:54.661794 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:54.661849 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:54.661873 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:54.661977 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:54.662213 (+ 236us) server_negotiation.cc:299] Negotiation successful
0621 07:55:54.662371 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":148,"threads_started":1}
I20260621 07:55:54.666290 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:55:54.666290 302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f26b46822e438abb594c541bff5be2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39973 } }
I20260621 07:55:54.666581 302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:54.666666 302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f26b46822e438abb594c541bff5be2, State: Initialized, Role: FOLLOWER
I20260621 07:55:54.667146 302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f26b46822e438abb594c541bff5be2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39973 } }
I20260621 07:55:54.667268 302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:55:54.667348 302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:55:54.667464 302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:54.668494 302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f26b46822e438abb594c541bff5be2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39973 } }
I20260621 07:55:54.668881 302 leader_election.cc:302] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59f26b46822e438abb594c541bff5be2; no voters:
I20260621 07:55:54.669229 302 leader_election.cc:288] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:55:54.669330 312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:54.669554 312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [term 1 LEADER]: Becoming Leader. State: Replica: 59f26b46822e438abb594c541bff5be2, State: Running, Role: LEADER
I20260621 07:55:54.669898 312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f26b46822e438abb594c541bff5be2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39973 } }
I20260621 07:55:54.670593 302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:55:54.671185 314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59f26b46822e438abb594c541bff5be2. Latest consensus state: current_term: 1 leader_uuid: "59f26b46822e438abb594c541bff5be2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f26b46822e438abb594c541bff5be2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39973 } } }
I20260621 07:55:54.671316 314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:54.671974 313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59f26b46822e438abb594c541bff5be2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f26b46822e438abb594c541bff5be2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39973 } } }
I20260621 07:55:54.672081 313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2 [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:54.673231 318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:55:54.673657 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:54.668558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36147 (local address 127.0.0.1:39973)
0621 07:55:54.668677 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:54.668681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:54.668892 (+ 211us) server_negotiation.cc:407] Connection header received
0621 07:55:54.668989 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:54.668992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:54.669041 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:54.669122 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:54.669439 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:54.670216 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:54.671393 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:54.671739 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:54.671838 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:54.673062 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:54.673066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:54.673069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:54.673084 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:54.673113 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:54.673127 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:54.673204 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:54.673471 (+ 267us) server_negotiation.cc:299] Negotiation successful
0621 07:55:54.673518 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:55:54.676849 318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:55:54.682792 318 catalog_manager.cc:1347] Generated new cluster ID: 47b0bc5ff4f24cf48b2b1770d1a1f38b
I20260621 07:55:54.682873 318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:55:54.697055 318 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:55:54.698457 318 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:55:54.707669 318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59f26b46822e438abb594c541bff5be2: Generated new TSK 0
I20260621 07:55:54.708640 318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:55:54.718568 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 32702
W20260621 07:55:54.724234 32687 connection.cc:535] client connection to 127.0.0.1:39973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39973): Connection reset by peer (error 104)
2026-06-21T07:55:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:55:54 dist-test-slave-lbpt krb5kdc[333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:55:54 dist-test-slave-lbpt krb5kdc[333](info): set up 2 sockets
Jun 21 07:55:54 dist-test-slave-lbpt krb5kdc[333](info): commencing operation
krb5kdc: starting...
W20260621 07:55:55.762632 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:55:55 dist-test-slave-lbpt krb5kdc[333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:55:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:55:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:55:55.891244 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:56.008126 346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:56.008437 346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:56.008482 346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:56.012751 346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:55:56.012822 346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:56.012843 346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:56.012861 346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:55:56.012876 346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:55:56.018806 346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37129
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:56.019956 346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:56.021016 346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:56.030117 354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:56.030143 352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:56.030133 346 server_base.cc:1034] running on GCE node
W20260621 07:55:56.030112 351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:56.030980 346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:56.032169 346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:56.033360 346 hybrid_clock.cc:648] HybridClock initialized: now 1782028556033329 us; error 47 us; skew 500 ppm
Jun 21 07:55:56 dist-test-slave-lbpt krb5kdc[333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:56.040251 346 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:55:56.041234 346 webserver.cc:460] Webserver started at http://127.0.0.1:44783/ using document root <none> and password file <none>
I20260621 07:55:56.041785 346 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:56.041846 346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:56.042017 346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:56.044133 346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "dddf13c8f1e5477ca20bae0cd6f06a43"
format_stamp: "Formatted at 2026-06-21 07:55:56 on dist-test-slave-lbpt"
server_key: "d3837954045df62254dff34240fb9142"
server_key_iv: "90f7763814c186cf6466661dfd53fc89"
server_key_version: "encryptionkey@0"
I20260621 07:55:56.044648 346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "dddf13c8f1e5477ca20bae0cd6f06a43"
format_stamp: "Formatted at 2026-06-21 07:55:56 on dist-test-slave-lbpt"
server_key: "d3837954045df62254dff34240fb9142"
server_key_iv: "90f7763814c186cf6466661dfd53fc89"
server_key_version: "encryptionkey@0"
I20260621 07:55:56.048506 346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260621 07:55:56.051041 361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:56.052443 346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:55:56.052562 346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "dddf13c8f1e5477ca20bae0cd6f06a43"
format_stamp: "Formatted at 2026-06-21 07:55:56 on dist-test-slave-lbpt"
server_key: "d3837954045df62254dff34240fb9142"
server_key_iv: "90f7763814c186cf6466661dfd53fc89"
server_key_version: "encryptionkey@0"
I20260621 07:55:56.052662 346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:56.057385 346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:56.061019 346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:56.061232 346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:56.068539 346 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37129
I20260621 07:55:56.068557 413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37129 every 8 connection(s)
I20260621 07:55:56.069507 346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:55:56.072489 414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:56.078150 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 346
I20260621 07:55:56.078305 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:55:56.078655 12336 external_mini_cluster.cc:1428] Setting key f9a9537e2e77dc087ef5d9686ad1bb68
I20260621 07:55:56.079078 414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43: Bootstrap starting.
I20260621 07:55:56.081982 414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:56.082937 414 log.cc:824] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:56.084863 417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:56.080241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53337 (local address 127.0.0.1:37129)
0621 07:55:56.080824 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:56.080839 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:56.080878 (+ 39us) server_negotiation.cc:407] Connection header received
0621 07:55:56.081582 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:56.081601 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:56.082013 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:56.082478 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:56.083276 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:56.083281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:56.083312 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:56.083327 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:56.083396 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:56.083452 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:56.083950 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:56.084013 (+ 63us) server_negotiation.cc:299] Negotiation successful
0621 07:55:56.084152 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":168,"threads_started":1}
I20260621 07:55:56.085198 414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43: No bootstrap required, opened a new log
I20260621 07:55:56.088279 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:55:56.089823 414 raft_consensus.cc:348] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddf13c8f1e5477ca20bae0cd6f06a43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37129 } }
I20260621 07:55:56.090111 414 raft_consensus.cc:374] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:56.090190 414 raft_consensus.cc:729] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dddf13c8f1e5477ca20bae0cd6f06a43, State: Initialized, Role: FOLLOWER
I20260621 07:55:56.090791 414 consensus_queue.cc:260] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddf13c8f1e5477ca20bae0cd6f06a43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37129 } }
I20260621 07:55:56.090973 414 raft_consensus.cc:388] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:55:56.091094 414 raft_consensus.cc:482] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:55:56.091217 414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:56.092149 417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:56.090392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54641 (local address 127.0.0.1:37129)
0621 07:55:56.090537 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:56.090542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:56.090760 (+ 218us) server_negotiation.cc:407] Connection header received
0621 07:55:56.090846 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:56.090849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:56.090887 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:56.090980 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:56.091790 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:56.091793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:56.091795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:56.091807 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:56.091838 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:56.091851 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:56.091947 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:56.091994 (+ 47us) server_negotiation.cc:299] Negotiation successful
0621 07:55:56.092031 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 07:55:56.092576 414 raft_consensus.cc:504] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddf13c8f1e5477ca20bae0cd6f06a43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37129 } }
I20260621 07:55:56.092954 414 leader_election.cc:302] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dddf13c8f1e5477ca20bae0cd6f06a43; no voters:
I20260621 07:55:56.093331 414 leader_election.cc:288] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:55:56.093451 424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:56.093694 424 raft_consensus.cc:686] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [term 1 LEADER]: Becoming Leader. State: Replica: dddf13c8f1e5477ca20bae0cd6f06a43, State: Running, Role: LEADER
I20260621 07:55:56.094108 424 consensus_queue.cc:237] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddf13c8f1e5477ca20bae0cd6f06a43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37129 } }
I20260621 07:55:56.094744 414 sys_catalog.cc:564] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:55:56.096097 426 sys_catalog.cc:455] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dddf13c8f1e5477ca20bae0cd6f06a43. Latest consensus state: current_term: 1 leader_uuid: "dddf13c8f1e5477ca20bae0cd6f06a43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddf13c8f1e5477ca20bae0cd6f06a43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37129 } } }
I20260621 07:55:56.096246 426 sys_catalog.cc:458] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:56.096129 425 sys_catalog.cc:455] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dddf13c8f1e5477ca20bae0cd6f06a43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddf13c8f1e5477ca20bae0cd6f06a43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37129 } } }
I20260621 07:55:56.096570 425 sys_catalog.cc:458] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43 [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:56.096657 433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:55:56.099706 433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:55:56.105813 433 catalog_manager.cc:1347] Generated new cluster ID: 0125ee23df664d9c8a135f4300f6052b
I20260621 07:55:56.105923 433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:55:56.117079 433 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:55:56.117430 433 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:55:56.124485 433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dddf13c8f1e5477ca20bae0cd6f06a43: Generated new TSK 0
I20260621 07:55:56.125255 433 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:55:56.133150 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 346
2026-06-21T07:55:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:55:56 dist-test-slave-lbpt krb5kdc[445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:55:56 dist-test-slave-lbpt krb5kdc[445](info): set up 2 sockets
Jun 21 07:55:56 dist-test-slave-lbpt krb5kdc[445](info): commencing operation
krb5kdc: starting...
W20260621 07:55:57.178692 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:55:57 dist-test-slave-lbpt krb5kdc[445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:55:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:55:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:55:57.307896 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:57.421777 458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:57.422119 458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:57.422163 458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:57.426290 458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:55:57.426364 458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:57.426383 458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:57.426400 458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:55:57.426467 458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:55:57.432155 458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:57.433306 458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:57.434466 458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:57.443495 464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:57.443455 466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:57.443482 463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:57.443743 458 server_base.cc:1034] running on GCE node
I20260621 07:55:57.444483 458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:57.445596 458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:57.446754 458 hybrid_clock.cc:648] HybridClock initialized: now 1782028557446738 us; error 32 us; skew 500 ppm
Jun 21 07:55:57 dist-test-slave-lbpt krb5kdc[445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:57.449549 458 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:55:57.450507 458 webserver.cc:460] Webserver started at http://127.0.0.1:43923/ using document root <none> and password file <none>
I20260621 07:55:57.451109 458 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:57.451174 458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:57.451347 458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:57.453557 458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "d843935636e949998440d237115bbe9a"
format_stamp: "Formatted at 2026-06-21 07:55:57 on dist-test-slave-lbpt"
server_key: "e76da1836b9599fe94aa1e02638233b9"
server_key_iv: "df5ba67d872a1716c30ddd8878b4e844"
server_key_version: "encryptionkey@0"
I20260621 07:55:57.454150 458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "d843935636e949998440d237115bbe9a"
format_stamp: "Formatted at 2026-06-21 07:55:57 on dist-test-slave-lbpt"
server_key: "e76da1836b9599fe94aa1e02638233b9"
server_key_iv: "df5ba67d872a1716c30ddd8878b4e844"
server_key_version: "encryptionkey@0"
I20260621 07:55:57.458109 458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:55:57.460748 473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:57.462227 458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 07:55:57.462345 458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "d843935636e949998440d237115bbe9a"
format_stamp: "Formatted at 2026-06-21 07:55:57 on dist-test-slave-lbpt"
server_key: "e76da1836b9599fe94aa1e02638233b9"
server_key_iv: "df5ba67d872a1716c30ddd8878b4e844"
server_key_version: "encryptionkey@0"
I20260621 07:55:57.462517 458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:57.481333 458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:57.484771 458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:57.484930 458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:57.491922 458 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40113
I20260621 07:55:57.491999 525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40113 every 8 connection(s)
I20260621 07:55:57.492940 458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:55:57.494710 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 458
I20260621 07:55:57.494861 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:55:57.495167 12336 external_mini_cluster.cc:1428] Setting key cd478ba941bfb3d4be80342849a81993
I20260621 07:55:57.496100 526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:57.503101 526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a: Bootstrap starting.
Jun 21 07:55:57 dist-test-slave-lbpt krb5kdc[445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 07:55:57.505604 526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:57.506562 526 log.cc:824] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:57.508869 526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a: No bootstrap required, opened a new log
I20260621 07:55:57.511171 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:57.496672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60485 (local address 127.0.0.1:40113)
0621 07:55:57.497109 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:57.497128 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:57.497171 (+ 43us) server_negotiation.cc:407] Connection header received
0621 07:55:57.498087 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:57.498111 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:57.498582 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:57.498966 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:57.499680 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:57.501241 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:57.502182 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:57.502890 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:57.503059 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:57.505877 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:57.505885 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:57.505904 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:57.505937 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:57.507906 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:57.508339 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:57.508345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:57.508350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:57.508486 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:57.508833 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:57.508836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:57.508837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:57.509357 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:57.509541 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:57.509818 (+ 277us) server_negotiation.cc:299] Negotiation successful
0621 07:55:57.510122 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":125,"threads_started":1}
I20260621 07:55:57.513440 526 raft_consensus.cc:348] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843935636e949998440d237115bbe9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40113 } }
I20260621 07:55:57.513720 526 raft_consensus.cc:374] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:57.513785 526 raft_consensus.cc:729] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d843935636e949998440d237115bbe9a, State: Initialized, Role: FOLLOWER
I20260621 07:55:57.514263 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:55:57.514273 526 consensus_queue.cc:260] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843935636e949998440d237115bbe9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40113 } }
I20260621 07:55:57.514451 526 raft_consensus.cc:388] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:55:57.514521 526 raft_consensus.cc:482] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:55:57.514639 526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:57.515626 526 raft_consensus.cc:504] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843935636e949998440d237115bbe9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40113 } }
I20260621 07:55:57.515988 526 leader_election.cc:302] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d843935636e949998440d237115bbe9a; no voters:
I20260621 07:55:57.516317 526 leader_election.cc:288] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:55:57.516482 535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:57.516826 535 raft_consensus.cc:686] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [term 1 LEADER]: Becoming Leader. State: Replica: d843935636e949998440d237115bbe9a, State: Running, Role: LEADER
I20260621 07:55:57.517113 535 consensus_queue.cc:237] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843935636e949998440d237115bbe9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40113 } }
I20260621 07:55:57.517584 526 sys_catalog.cc:564] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:55:57.518885 537 sys_catalog.cc:455] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d843935636e949998440d237115bbe9a. Latest consensus state: current_term: 1 leader_uuid: "d843935636e949998440d237115bbe9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843935636e949998440d237115bbe9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40113 } } }
I20260621 07:55:57.518894 536 sys_catalog.cc:455] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d843935636e949998440d237115bbe9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843935636e949998440d237115bbe9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40113 } } }
I20260621 07:55:57.519034 536 sys_catalog.cc:458] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:57.519034 537 sys_catalog.cc:458] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:57.519572 546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:55:57.523201 546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:55:57.529459 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:57.520222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36167 (local address 127.0.0.1:40113)
0621 07:55:57.520354 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:57.520361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:57.520656 (+ 295us) server_negotiation.cc:407] Connection header received
0621 07:55:57.520796 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:57.520799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:57.520848 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:57.520920 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:57.521798 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:57.522623 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:57.523766 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:57.524121 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:57.524238 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:57.525757 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:57.525761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:57.525764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:55:57.525797 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:57.527385 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:57.528035 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:57.528039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:57.528041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:57.528140 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:55:57.528464 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:55:57.528468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:55:57.528470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:55:57.528705 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:57.528836 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:57.529170 (+ 334us) server_negotiation.cc:299] Negotiation successful
0621 07:55:57.529307 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 07:55:57.530241 546 catalog_manager.cc:1347] Generated new cluster ID: c5cf7f3fb24c46d89f60fd53994f57a5
I20260621 07:55:57.530362 546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:55:57.541590 546 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:55:57.542695 546 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:55:57.546922 546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d843935636e949998440d237115bbe9a: Generated new TSK 0
I20260621 07:55:57.547608 546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:55:57.553933 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 458
2026-06-21T07:55:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:55:57 dist-test-slave-lbpt krb5kdc[557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:55:57 dist-test-slave-lbpt krb5kdc[557](info): set up 2 sockets
Jun 21 07:55:57 dist-test-slave-lbpt krb5kdc[557](info): commencing operation
krb5kdc: starting...
W20260621 07:55:58.595439 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:55:58 dist-test-slave-lbpt krb5kdc[557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:55:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:55:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:55:58.721547 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:55:58.832607 570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:55:58.832904 570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:55:58.832947 570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:55:58.837045 570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:55:58.837107 570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:55:58.837126 570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:55:58.837143 570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:55:58.837158 570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:55:58.842430 570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:55:58.843474 570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:55:58.844452 570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:55:58.853127 578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:55:58.853144 575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:58.853227 570 server_base.cc:1034] running on GCE node
W20260621 07:55:58.853150 576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:55:58.853943 570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:55:58.855131 570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:55:58.856364 570 hybrid_clock.cc:648] HybridClock initialized: now 1782028558856354 us; error 30 us; skew 500 ppm
Jun 21 07:55:58 dist-test-slave-lbpt krb5kdc[557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:55:58.859232 570 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:55:58.860124 570 webserver.cc:460] Webserver started at http://127.0.0.1:43901/ using document root <none> and password file <none>
I20260621 07:55:58.860703 570 fs_manager.cc:359] Metadata directory not provided
I20260621 07:55:58.860795 570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:55:58.861034 570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:55:58.863171 570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "7511eddf6d3e4db28139e192cb27365c"
format_stamp: "Formatted at 2026-06-21 07:55:58 on dist-test-slave-lbpt"
server_key: "889213bbc6fb0ee5839a0a3d675cdb2d"
server_key_iv: "ca7874676bba98d4a7c42eff8a8fc1ca"
server_key_version: "encryptionkey@0"
I20260621 07:55:58.863749 570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "7511eddf6d3e4db28139e192cb27365c"
format_stamp: "Formatted at 2026-06-21 07:55:58 on dist-test-slave-lbpt"
server_key: "889213bbc6fb0ee5839a0a3d675cdb2d"
server_key_iv: "ca7874676bba98d4a7c42eff8a8fc1ca"
server_key_version: "encryptionkey@0"
I20260621 07:55:58.867529 570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:55:58.869949 585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:55:58.871321 570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:55:58.871465 570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "7511eddf6d3e4db28139e192cb27365c"
format_stamp: "Formatted at 2026-06-21 07:55:58 on dist-test-slave-lbpt"
server_key: "889213bbc6fb0ee5839a0a3d675cdb2d"
server_key_iv: "ca7874676bba98d4a7c42eff8a8fc1ca"
server_key_version: "encryptionkey@0"
I20260621 07:55:58.871582 570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:55:58.902946 570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:55:58.906200 570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:55:58.906492 570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:55:58.913673 570 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34845
I20260621 07:55:58.913694 637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34845 every 8 connection(s)
I20260621 07:55:58.914713 570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:55:58.917480 638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:55:58.918404 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 570
I20260621 07:55:58.918581 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:55:58.918911 12336 external_mini_cluster.cc:1428] Setting key a2b83991ecd124cfa9b020174d76f107
I20260621 07:55:58.923763 638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c: Bootstrap starting.
I20260621 07:55:58.926040 638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c: Neither blocks nor log segments found. Creating new log.
I20260621 07:55:58.926844 638 log.cc:824] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:55:58.927975 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:58.920357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41059 (local address 127.0.0.1:34845)
0621 07:55:58.920810 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:58.920824 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:58.920859 (+ 35us) server_negotiation.cc:407] Connection header received
0621 07:55:58.921512 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:58.921528 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:58.921865 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:58.922212 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:58.922642 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:58.923600 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:58.924552 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:58.925026 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:58.925199 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:58.926059 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:58.926064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:58.926077 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:58.926564 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:58.926619 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:58.926643 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:58.926758 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:58.926909 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 07:55:58.927109 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":138,"threads_started":1}
I20260621 07:55:58.929036 638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c: No bootstrap required, opened a new log
I20260621 07:55:58.930842 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:55:58.932971 638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7511eddf6d3e4db28139e192cb27365c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } }
I20260621 07:55:58.933283 638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:55:58.933369 638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7511eddf6d3e4db28139e192cb27365c, State: Initialized, Role: FOLLOWER
I20260621 07:55:58.933885 638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7511eddf6d3e4db28139e192cb27365c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } }
I20260621 07:55:58.934049 638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:55:58.934139 638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:55:58.934250 638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:55:58.935267 638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7511eddf6d3e4db28139e192cb27365c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } }
I20260621 07:55:58.935664 638 leader_election.cc:302] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7511eddf6d3e4db28139e192cb27365c; no voters:
I20260621 07:55:58.936022 638 leader_election.cc:288] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:55:58.936094 648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:55:58.936487 648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [term 1 LEADER]: Becoming Leader. State: Replica: 7511eddf6d3e4db28139e192cb27365c, State: Running, Role: LEADER
I20260621 07:55:58.936844 648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7511eddf6d3e4db28139e192cb27365c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } }
I20260621 07:55:58.937366 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:55:58.932801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39857 (local address 127.0.0.1:34845)
0621 07:55:58.932932 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 07:55:58.932936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:55:58.933123 (+ 187us) server_negotiation.cc:407] Connection header received
0621 07:55:58.933260 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:55:58.933266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:55:58.933323 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:55:58.933416 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:55:58.933713 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:58.934402 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:58.935344 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:55:58.935783 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:55:58.935976 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:55:58.936876 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:55:58.936880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:55:58.936883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:55:58.936895 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:55:58.936922 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:55:58.936935 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:55:58.937001 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 07:55:58.937163 (+ 162us) server_negotiation.cc:299] Negotiation successful
0621 07:55:58.937218 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:55:58.937708 638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:55:58.939082 649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7511eddf6d3e4db28139e192cb27365c. Latest consensus state: current_term: 1 leader_uuid: "7511eddf6d3e4db28139e192cb27365c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7511eddf6d3e4db28139e192cb27365c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } } }
I20260621 07:55:58.939083 650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7511eddf6d3e4db28139e192cb27365c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7511eddf6d3e4db28139e192cb27365c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } } }
I20260621 07:55:58.939268 650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:58.939545 649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c [sys.catalog]: This master's current role is: LEADER
I20260621 07:55:58.939745 658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:55:58.942607 658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:55:58.947844 658 catalog_manager.cc:1347] Generated new cluster ID: d2fccdbb3a52488698385fa74c7b7d8f
I20260621 07:55:58.947922 658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:55:58.969756 658 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:55:58.971200 658 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:55:58.977613 658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7511eddf6d3e4db28139e192cb27365c: Generated new TSK 0
I20260621 07:55:58.978513 658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:55:58.986037 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 570
2026-06-21T07:55:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:55:59 dist-test-slave-lbpt krb5kdc[669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:55:59 dist-test-slave-lbpt krb5kdc[669](info): set up 2 sockets
Jun 21 07:55:59 dist-test-slave-lbpt krb5kdc[669](info): commencing operation
krb5kdc: starting...
W20260621 07:56:00.046886 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:56:00 dist-test-slave-lbpt krb5kdc[669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:56:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:56:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:56:00.171599 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:00.293180 682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:00.293558 682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:00.293649 682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:00.299662 682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:00.299749 682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:00.299785 682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:00.299830 682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:00.299854 682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:00.310134 682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:00.311781 682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:00.313280 682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:00.323727 687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:00.323843 682 server_base.cc:1034] running on GCE node
W20260621 07:56:00.323737 688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:00.323727 690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:00.324642 682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:00.325711 682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:00.326889 682 hybrid_clock.cc:648] HybridClock initialized: now 1782028560326854 us; error 46 us; skew 500 ppm
Jun 21 07:56:00 dist-test-slave-lbpt krb5kdc[669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:00.330054 682 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:56:00.331076 682 webserver.cc:460] Webserver started at http://127.0.0.1:33275/ using document root <none> and password file <none>
I20260621 07:56:00.331622 682 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:00.331684 682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:00.331859 682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:00.333999 682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "ee69f4d4c8204e5daa71fe1fe767777c"
format_stamp: "Formatted at 2026-06-21 07:56:00 on dist-test-slave-lbpt"
server_key: "6a12d4e69ff46fda7285511df3079b2e"
server_key_iv: "ef887406fd1f98d15db119217add8626"
server_key_version: "encryptionkey@0"
I20260621 07:56:00.334689 682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "ee69f4d4c8204e5daa71fe1fe767777c"
format_stamp: "Formatted at 2026-06-21 07:56:00 on dist-test-slave-lbpt"
server_key: "6a12d4e69ff46fda7285511df3079b2e"
server_key_iv: "ef887406fd1f98d15db119217add8626"
server_key_version: "encryptionkey@0"
I20260621 07:56:00.338682 682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:56:00.341289 697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:00.342788 682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:56:00.342904 682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "ee69f4d4c8204e5daa71fe1fe767777c"
format_stamp: "Formatted at 2026-06-21 07:56:00 on dist-test-slave-lbpt"
server_key: "6a12d4e69ff46fda7285511df3079b2e"
server_key_iv: "ef887406fd1f98d15db119217add8626"
server_key_version: "encryptionkey@0"
I20260621 07:56:00.343050 682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:00.347911 682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:00.351075 682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:00.351281 682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:00.358489 682 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38317
I20260621 07:56:00.358521 749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38317 every 8 connection(s)
I20260621 07:56:00.359781 682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:56:00.362636 750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:00.368336 750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c: Bootstrap starting.
I20260621 07:56:00.368557 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 682
I20260621 07:56:00.368705 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:56:00.369004 12336 external_mini_cluster.cc:1428] Setting key 4038feccb5de45f058af7b37d92db104
I20260621 07:56:00.371107 750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:00.371862 750 log.cc:824] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:00.374105 750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c: No bootstrap required, opened a new log
I20260621 07:56:00.374470 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:00.370466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43041 (local address 127.0.0.1:38317)
0621 07:56:00.370923 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:00.370939 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:00.370974 (+ 35us) server_negotiation.cc:407] Connection header received
0621 07:56:00.371671 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:00.371690 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:00.372033 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:00.372395 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:00.373187 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:00.373192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:00.373203 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:00.373213 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:00.373265 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:00.373283 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:00.373408 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:00.373518 (+ 110us) server_negotiation.cc:299] Negotiation successful
0621 07:56:00.373676 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":144,"threads_started":1}
I20260621 07:56:00.377588 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:56:00.378468 750 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee69f4d4c8204e5daa71fe1fe767777c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38317 } }
I20260621 07:56:00.378696 750 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:56:00.378782 750 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee69f4d4c8204e5daa71fe1fe767777c, State: Initialized, Role: FOLLOWER
I20260621 07:56:00.379236 750 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee69f4d4c8204e5daa71fe1fe767777c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38317 } }
I20260621 07:56:00.379402 750 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:56:00.379487 750 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:56:00.379594 750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:00.380908 750 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee69f4d4c8204e5daa71fe1fe767777c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38317 } }
I20260621 07:56:00.381318 750 leader_election.cc:302] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee69f4d4c8204e5daa71fe1fe767777c; no voters:
I20260621 07:56:00.381560 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:00.379829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39903 (local address 127.0.0.1:38317)
0621 07:56:00.379957 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:00.379961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:00.380139 (+ 178us) server_negotiation.cc:407] Connection header received
0621 07:56:00.380250 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:00.380252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:00.380282 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:00.380347 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:00.381119 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:00.381123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:00.381126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:00.381138 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:00.381172 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:00.381183 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:00.381261 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:00.381379 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 07:56:00.381413 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:56:00.381678 750 leader_election.cc:288] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:56:00.381981 760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:56:00.382200 760 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [term 1 LEADER]: Becoming Leader. State: Replica: ee69f4d4c8204e5daa71fe1fe767777c, State: Running, Role: LEADER
I20260621 07:56:00.382601 760 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee69f4d4c8204e5daa71fe1fe767777c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38317 } }
I20260621 07:56:00.383105 750 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:00.384418 762 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee69f4d4c8204e5daa71fe1fe767777c. Latest consensus state: current_term: 1 leader_uuid: "ee69f4d4c8204e5daa71fe1fe767777c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee69f4d4c8204e5daa71fe1fe767777c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38317 } } }
I20260621 07:56:00.384429 761 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee69f4d4c8204e5daa71fe1fe767777c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee69f4d4c8204e5daa71fe1fe767777c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38317 } } }
I20260621 07:56:00.384547 762 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:00.384547 761 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c [sys.catalog]: This master's current role is: LEADER
W20260621 07:56:00.387984 775 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:56:00.388080 775 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:56:00.388196 770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:56:00.389111 770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:56:00.395164 770 catalog_manager.cc:1347] Generated new cluster ID: a5cfa586afb34244bcd5e3c365806666
I20260621 07:56:00.395238 770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:56:00.405117 770 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:56:00.405478 770 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:56:00.418942 770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee69f4d4c8204e5daa71fe1fe767777c: Generated new TSK 0
I20260621 07:56:00.419819 770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:56:00.428167 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 682
2026-06-21T07:56:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:56:00 dist-test-slave-lbpt krb5kdc[781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:56:00 dist-test-slave-lbpt krb5kdc[781](info): set up 2 sockets
Jun 21 07:56:00 dist-test-slave-lbpt krb5kdc[781](info): commencing operation
krb5kdc: starting...
W20260621 07:56:01.462857 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:56:01 dist-test-slave-lbpt krb5kdc[781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:56:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:56:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:56:01.594698 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36517
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:01.707165 794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:01.707484 794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:01.707557 794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:01.711733 794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:01.711843 794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:01.711882 794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:01.711922 794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:01.711963 794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:01.717465 794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:01.718709 794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:01.719762 794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:01.728549 799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:01.728534 802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:01.728579 800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:01.728695 794 server_base.cc:1034] running on GCE node
I20260621 07:56:01.729439 794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:01.730635 794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:01.731850 794 hybrid_clock.cc:648] HybridClock initialized: now 1782028561731831 us; error 34 us; skew 500 ppm
Jun 21 07:56:01 dist-test-slave-lbpt krb5kdc[781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:01.739372 794 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:56:01.740514 794 webserver.cc:460] Webserver started at http://127.0.0.1:43147/ using document root <none> and password file <none>
I20260621 07:56:01.741125 794 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:01.741223 794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:01.741511 794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:01.744103 794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "46610521cdab4d3c835d4b8b3e812fd8"
format_stamp: "Formatted at 2026-06-21 07:56:01 on dist-test-slave-lbpt"
server_key: "830a526f5d8c7573e0088f5ea2e50b06"
server_key_iv: "23e4dc8e5852deaf7390c4fae17dbdc7"
server_key_version: "encryptionkey@0"
I20260621 07:56:01.744724 794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "46610521cdab4d3c835d4b8b3e812fd8"
format_stamp: "Formatted at 2026-06-21 07:56:01 on dist-test-slave-lbpt"
server_key: "830a526f5d8c7573e0088f5ea2e50b06"
server_key_iv: "23e4dc8e5852deaf7390c4fae17dbdc7"
server_key_version: "encryptionkey@0"
I20260621 07:56:01.748924 794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:56:01.751668 809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:01.753146 794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:56:01.753322 794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "46610521cdab4d3c835d4b8b3e812fd8"
format_stamp: "Formatted at 2026-06-21 07:56:01 on dist-test-slave-lbpt"
server_key: "830a526f5d8c7573e0088f5ea2e50b06"
server_key_iv: "23e4dc8e5852deaf7390c4fae17dbdc7"
server_key_version: "encryptionkey@0"
I20260621 07:56:01.753453 794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:01.774751 794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:01.778195 794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:01.778453 794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:01.785480 794 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36517
I20260621 07:56:01.785568 861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36517 every 8 connection(s)
I20260621 07:56:01.786512 794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:56:01.789373 862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:01.791424 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 794
I20260621 07:56:01.791594 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:56:01.791905 12336 external_mini_cluster.cc:1428] Setting key a920784577a65f59ca22a57488cf212c
I20260621 07:56:01.796346 862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8: Bootstrap starting.
I20260621 07:56:01.798879 862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:01.800053 862 log.cc:824] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:56:01 dist-test-slave-lbpt krb5kdc[781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 07:56:01.802284 862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8: No bootstrap required, opened a new log
I20260621 07:56:01.805824 862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46610521cdab4d3c835d4b8b3e812fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } }
I20260621 07:56:01.806082 862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:56:01.806171 862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46610521cdab4d3c835d4b8b3e812fd8, State: Initialized, Role: FOLLOWER
I20260621 07:56:01.806711 862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46610521cdab4d3c835d4b8b3e812fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } }
I20260621 07:56:01.806883 862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:56:01.806982 862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:56:01.807103 862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:01.807518 865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:01.793273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:49749 (local address 127.0.0.1:36517)
0621 07:56:01.793678 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:01.793694 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:01.793730 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:56:01.794382 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:01.794399 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:01.794814 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:01.795093 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:01.796098 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:01.797263 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:01.798528 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:01.799050 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:01.799266 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:01.802285 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:01.802290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:01.802304 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:01.802354 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:01.804221 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:01.804772 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:01.804826 (+ 54us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:01.804833 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:01.804943 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:01.805295 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:01.805299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:01.805302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:01.805724 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:01.805957 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:01.806280 (+ 323us) server_negotiation.cc:299] Negotiation successful
0621 07:56:01.806578 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":139,"threads_started":1}
I20260621 07:56:01.808141 862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46610521cdab4d3c835d4b8b3e812fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } }
I20260621 07:56:01.808717 862 leader_election.cc:302] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46610521cdab4d3c835d4b8b3e812fd8; no voters:
I20260621 07:56:01.809118 862 leader_election.cc:288] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:56:01.809216 867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:56:01.809419 867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [term 1 LEADER]: Becoming Leader. State: Replica: 46610521cdab4d3c835d4b8b3e812fd8, State: Running, Role: LEADER
I20260621 07:56:01.809805 867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46610521cdab4d3c835d4b8b3e812fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } }
I20260621 07:56:01.810537 862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:01.810925 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:56:01.811141 867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46610521cdab4d3c835d4b8b3e812fd8. Latest consensus state: current_term: 1 leader_uuid: "46610521cdab4d3c835d4b8b3e812fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46610521cdab4d3c835d4b8b3e812fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } } }
I20260621 07:56:01.811249 867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:01.813560 868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46610521cdab4d3c835d4b8b3e812fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46610521cdab4d3c835d4b8b3e812fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } } }
I20260621 07:56:01.813838 868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8 [sys.catalog]: This master's current role is: LEADER
W20260621 07:56:01.816231 887 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:56:01.816336 887 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:56:01.816430 872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:56:01.817207 872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:56:01.823202 872 catalog_manager.cc:1347] Generated new cluster ID: 64be4c3b23704dfb90f42ba1485ed387
I20260621 07:56:01.823264 872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:56:01.823418 865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:01.813589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:33131 (local address 127.0.0.1:36517)
0621 07:56:01.813707 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:01.813711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:01.813959 (+ 248us) server_negotiation.cc:407] Connection header received
0621 07:56:01.814115 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:01.814118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:01.814167 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:01.814251 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:01.815023 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:01.815771 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:01.816892 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:01.817208 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:01.817316 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:01.819128 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:01.819132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:01.819135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:01.819160 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:01.820900 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:01.821930 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:01.821933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:01.821935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:01.822023 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:01.822401 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:01.822405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:01.822407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:01.822639 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:01.822788 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:01.823145 (+ 357us) server_negotiation.cc:299] Negotiation successful
0621 07:56:01.823252 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:56:01.844630 872 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:56:01.845600 872 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:56:01.854053 872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46610521cdab4d3c835d4b8b3e812fd8: Generated new TSK 0
I20260621 07:56:01.854839 872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:56:01.862957 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 794
W20260621 07:56:01.868427 779 connection.cc:535] client connection to 127.0.0.1:36517 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36517): Connection reset by peer (error 104)
2026-06-21T07:56:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:56:01 dist-test-slave-lbpt krb5kdc[893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:56:01 dist-test-slave-lbpt krb5kdc[893](info): set up 2 sockets
Jun 21 07:56:01 dist-test-slave-lbpt krb5kdc[893](info): commencing operation
krb5kdc: starting...
W20260621 07:56:02.910826 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:56:02 dist-test-slave-lbpt krb5kdc[893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:56:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:56:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:56:03.039606 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36993
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:03.154624 906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:03.154987 906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:03.155074 906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:03.159236 906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:03.159353 906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:03.159392 906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:03.159454 906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:03.159494 906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:03.164988 906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:03.166180 906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:03.167320 906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:03.176189 914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:03.176172 912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:03.176383 911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:03.176822 906 server_base.cc:1034] running on GCE node
I20260621 07:56:03.177407 906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:03.178611 906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:03.179806 906 hybrid_clock.cc:648] HybridClock initialized: now 1782028563179783 us; error 31 us; skew 500 ppm
Jun 21 07:56:03 dist-test-slave-lbpt krb5kdc[893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:03.182744 906 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:56:03.183641 906 webserver.cc:460] Webserver started at http://127.0.0.1:44183/ using document root <none> and password file <none>
I20260621 07:56:03.184201 906 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:03.184262 906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:03.184439 906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:03.186617 906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "d6c29b0e5ed341e690a6df3aa15e97a3"
format_stamp: "Formatted at 2026-06-21 07:56:03 on dist-test-slave-lbpt"
server_key: "7589a81f710e52ed19f6b65ae60ff34a"
server_key_iv: "544fa48c03bca481b1a249df7f98607f"
server_key_version: "encryptionkey@0"
I20260621 07:56:03.187165 906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "d6c29b0e5ed341e690a6df3aa15e97a3"
format_stamp: "Formatted at 2026-06-21 07:56:03 on dist-test-slave-lbpt"
server_key: "7589a81f710e52ed19f6b65ae60ff34a"
server_key_iv: "544fa48c03bca481b1a249df7f98607f"
server_key_version: "encryptionkey@0"
I20260621 07:56:03.191538 906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 07:56:03.194096 921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:03.195564 906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:56:03.195734 906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "d6c29b0e5ed341e690a6df3aa15e97a3"
format_stamp: "Formatted at 2026-06-21 07:56:03 on dist-test-slave-lbpt"
server_key: "7589a81f710e52ed19f6b65ae60ff34a"
server_key_iv: "544fa48c03bca481b1a249df7f98607f"
server_key_version: "encryptionkey@0"
I20260621 07:56:03.195884 906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:03.207800 906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:03.211398 906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:03.211625 906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:03.219477 906 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36993
I20260621 07:56:03.219502 973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36993 every 8 connection(s)
I20260621 07:56:03.220566 906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:56:03.223623 974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:03.226595 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 906
I20260621 07:56:03.226753 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:56:03.227082 12336 external_mini_cluster.cc:1428] Setting key 5fa382355b2478c733dc9c70cc25d960
I20260621 07:56:03.230125 974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3: Bootstrap starting.
I20260621 07:56:03.232844 974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:03.233666 974 log.cc:824] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:03.236049 974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3: No bootstrap required, opened a new log
I20260621 07:56:03.236948 977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:03.228617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:59585 (local address 127.0.0.1:36993)
0621 07:56:03.229075 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:03.229098 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:03.229135 (+ 37us) server_negotiation.cc:407] Connection header received
0621 07:56:03.229825 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:03.229843 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:03.230264 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:03.230641 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:03.230997 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:03.231973 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:03.233192 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:03.233704 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:03.233905 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:03.235037 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:03.235042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:03.235056 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:03.235484 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:03.235539 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:03.235562 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:03.235670 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:03.236015 (+ 345us) server_negotiation.cc:299] Negotiation successful
0621 07:56:03.236182 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":145,"threads_started":1}
I20260621 07:56:03.240247 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:56:03.240338 974 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c29b0e5ed341e690a6df3aa15e97a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36993 } }
I20260621 07:56:03.240608 974 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:56:03.240684 974 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c29b0e5ed341e690a6df3aa15e97a3, State: Initialized, Role: FOLLOWER
I20260621 07:56:03.241156 974 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c29b0e5ed341e690a6df3aa15e97a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36993 } }
I20260621 07:56:03.241307 974 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:56:03.241391 974 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:56:03.241490 974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:03.242548 974 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c29b0e5ed341e690a6df3aa15e97a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36993 } }
I20260621 07:56:03.243033 974 leader_election.cc:302] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6c29b0e5ed341e690a6df3aa15e97a3; no voters:
I20260621 07:56:03.243527 974 leader_election.cc:288] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:56:03.243628 984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:56:03.243883 984 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [term 1 LEADER]: Becoming Leader. State: Replica: d6c29b0e5ed341e690a6df3aa15e97a3, State: Running, Role: LEADER
I20260621 07:56:03.244254 984 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c29b0e5ed341e690a6df3aa15e97a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36993 } }
I20260621 07:56:03.244902 974 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:03.245632 986 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6c29b0e5ed341e690a6df3aa15e97a3. Latest consensus state: current_term: 1 leader_uuid: "d6c29b0e5ed341e690a6df3aa15e97a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c29b0e5ed341e690a6df3aa15e97a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36993 } } }
I20260621 07:56:03.245785 986 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:03.246117 985 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6c29b0e5ed341e690a6df3aa15e97a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c29b0e5ed341e690a6df3aa15e97a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36993 } } }
I20260621 07:56:03.246201 985 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:03.248102 977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:03.242692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:42005 (local address 127.0.0.1:36993)
0621 07:56:03.242821 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:03.242826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:03.243021 (+ 195us) server_negotiation.cc:407] Connection header received
0621 07:56:03.243137 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:03.243139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:03.243224 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:03.243322 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:03.243807 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:03.244587 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:03.245799 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:03.246153 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:03.246253 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:03.247593 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:03.247597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:03.247599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:03.247611 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:03.247634 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:03.247644 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:03.247709 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:03.247910 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 07:56:03.247945 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":39}
W20260621 07:56:03.250869 999 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:56:03.250967 999 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:56:03.251044 993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:56:03.251744 993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:56:03.257416 993 catalog_manager.cc:1347] Generated new cluster ID: 17fb08d6a5ae460ca8537ce6e91684e8
I20260621 07:56:03.257498 993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:56:03.267931 993 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:56:03.269286 993 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:56:03.278403 993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6c29b0e5ed341e690a6df3aa15e97a3: Generated new TSK 0
I20260621 07:56:03.279295 993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:56:03.286399 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 906
W20260621 07:56:03.292071 891 connection.cc:535] client connection to 127.0.0.1:36993 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36993): Connection reset by peer (error 104)
2026-06-21T07:56:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:56:03 dist-test-slave-lbpt krb5kdc[1005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:56:03 dist-test-slave-lbpt krb5kdc[1005](info): set up 2 sockets
Jun 21 07:56:03 dist-test-slave-lbpt krb5kdc[1005](info): commencing operation
krb5kdc: starting...
W20260621 07:56:04.334853 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:56:04 dist-test-slave-lbpt krb5kdc[1005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:56:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:56:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:56:04.457886 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46051
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:04.572139 1018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:04.572432 1018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:04.572476 1018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:04.576624 1018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:04.576695 1018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:04.576714 1018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:04.576731 1018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:04.576771 1018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:04.582247 1018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:04.583478 1018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:04.584558 1018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:04.593518 1024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:04.593500 1026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:04.593487 1023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:04.593495 1018 server_base.cc:1034] running on GCE node
I20260621 07:56:04.594322 1018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:04.595489 1018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:04.596688 1018 hybrid_clock.cc:648] HybridClock initialized: now 1782028564596626 us; error 77 us; skew 500 ppm
Jun 21 07:56:04 dist-test-slave-lbpt krb5kdc[1005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:04.599722 1018 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:56:04.600643 1018 webserver.cc:460] Webserver started at http://127.0.0.1:39453/ using document root <none> and password file <none>
I20260621 07:56:04.601171 1018 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:04.601228 1018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:04.601392 1018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:04.603461 1018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "8ce2e709d1874302bfd0119f88e65950"
format_stamp: "Formatted at 2026-06-21 07:56:04 on dist-test-slave-lbpt"
server_key: "48826bb05e5213e4409de87658572d2c"
server_key_iv: "cb7ce137e38911a3e25f96a4d628c187"
server_key_version: "encryptionkey@0"
I20260621 07:56:04.603979 1018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "8ce2e709d1874302bfd0119f88e65950"
format_stamp: "Formatted at 2026-06-21 07:56:04 on dist-test-slave-lbpt"
server_key: "48826bb05e5213e4409de87658572d2c"
server_key_iv: "cb7ce137e38911a3e25f96a4d628c187"
server_key_version: "encryptionkey@0"
I20260621 07:56:04.607877 1018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:56:04.610666 1033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:04.612071 1018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:56:04.612216 1018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "8ce2e709d1874302bfd0119f88e65950"
format_stamp: "Formatted at 2026-06-21 07:56:04 on dist-test-slave-lbpt"
server_key: "48826bb05e5213e4409de87658572d2c"
server_key_iv: "cb7ce137e38911a3e25f96a4d628c187"
server_key_version: "encryptionkey@0"
I20260621 07:56:04.612390 1018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:04.617153 1018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:04.620411 1018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:04.620613 1018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:04.628367 1018 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46051
I20260621 07:56:04.628392 1085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46051 every 8 connection(s)
I20260621 07:56:04.629338 1018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:56:04.632125 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:04.634716 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 1018
I20260621 07:56:04.634884 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:56:04.635177 12336 external_mini_cluster.cc:1428] Setting key 62a8419a747839ce6ab7c25c727d0706
I20260621 07:56:04.638217 1086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950: Bootstrap starting.
I20260621 07:56:04.640518 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:04.636676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:47045 (local address 127.0.0.1:46051)
0621 07:56:04.637105 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:04.637122 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:04.637156 (+ 34us) server_negotiation.cc:407] Connection header received
0621 07:56:04.637814 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:04.637832 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:04.638140 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:04.638537 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:04.639383 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:04.639387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:04.639398 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:04.639408 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:04.639460 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:04.639478 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:04.639621 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:04.639692 (+ 71us) server_negotiation.cc:299] Negotiation successful
0621 07:56:04.639858 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":124,"threads_started":1}
I20260621 07:56:04.640656 1086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:04.641427 1086 log.cc:824] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:04.643461 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:56:04.643646 1086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950: No bootstrap required, opened a new log
I20260621 07:56:04.647425 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:04.645570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:48379 (local address 127.0.0.1:46051)
0621 07:56:04.645683 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:04.645687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:04.645903 (+ 216us) server_negotiation.cc:407] Connection header received
0621 07:56:04.646048 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:04.646051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:04.646087 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:04.646168 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:04.647003 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:04.647007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:04.647010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:04.647021 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:04.647048 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:04.647061 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:04.647156 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:04.647250 (+ 94us) server_negotiation.cc:299] Negotiation successful
0621 07:56:04.647297 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 07:56:04.647746 1086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce2e709d1874302bfd0119f88e65950" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46051 } }
I20260621 07:56:04.648149 1086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:56:04.648257 1086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce2e709d1874302bfd0119f88e65950, State: Initialized, Role: FOLLOWER
I20260621 07:56:04.648761 1086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce2e709d1874302bfd0119f88e65950" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46051 } }
I20260621 07:56:04.648947 1086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:56:04.649040 1086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:56:04.649151 1086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:04.650096 1086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce2e709d1874302bfd0119f88e65950" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46051 } }
I20260621 07:56:04.650511 1086 leader_election.cc:302] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ce2e709d1874302bfd0119f88e65950; no voters:
I20260621 07:56:04.650862 1086 leader_election.cc:288] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:56:04.650992 1096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:56:04.651261 1096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [term 1 LEADER]: Becoming Leader. State: Replica: 8ce2e709d1874302bfd0119f88e65950, State: Running, Role: LEADER
I20260621 07:56:04.651715 1096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce2e709d1874302bfd0119f88e65950" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46051 } }
I20260621 07:56:04.652185 1086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:04.653524 1097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ce2e709d1874302bfd0119f88e65950" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce2e709d1874302bfd0119f88e65950" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46051 } } }
I20260621 07:56:04.653652 1097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:04.654024 1098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ce2e709d1874302bfd0119f88e65950. Latest consensus state: current_term: 1 leader_uuid: "8ce2e709d1874302bfd0119f88e65950" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce2e709d1874302bfd0119f88e65950" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46051 } } }
I20260621 07:56:04.654110 1098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950 [sys.catalog]: This master's current role is: LEADER
W20260621 07:56:04.656805 1111 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:56:04.656872 1111 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:56:04.656952 1107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:56:04.657636 1107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:56:04.663640 1107 catalog_manager.cc:1347] Generated new cluster ID: 5802ca8657234d08b51f9e991255ce5f
I20260621 07:56:04.663715 1107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:56:04.679979 1107 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:56:04.680349 1107 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:56:04.701941 1107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ce2e709d1874302bfd0119f88e65950: Generated new TSK 0
I20260621 07:56:04.702785 1107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:56:04.705543 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 1018
2026-06-21T07:56:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:56:04 dist-test-slave-lbpt krb5kdc[1117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:56:04 dist-test-slave-lbpt krb5kdc[1117](info): set up 2 sockets
Jun 21 07:56:04 dist-test-slave-lbpt krb5kdc[1117](info): commencing operation
krb5kdc: starting...
W20260621 07:56:05.746873 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:56:05 dist-test-slave-lbpt krb5kdc[1117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:56:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:56:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:56:05.873396 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37743
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:05.988893 1130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:05.989288 1130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:05.989377 1130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:05.993750 1130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:05.993849 1130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:05.993889 1130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:05.993937 1130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:05.993975 1130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:06.001773 1130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37743
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:06.003253 1130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:06.004344 1130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:06.013456 1136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:06.013450 1135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:06.013456 1138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:06.013844 1130 server_base.cc:1034] running on GCE node
I20260621 07:56:06.014559 1130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:06.015739 1130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:06.016945 1130 hybrid_clock.cc:648] HybridClock initialized: now 1782028566016923 us; error 35 us; skew 500 ppm
Jun 21 07:56:06 dist-test-slave-lbpt krb5kdc[1117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:06.020254 1130 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:56:06.021275 1130 webserver.cc:460] Webserver started at http://127.0.0.1:42399/ using document root <none> and password file <none>
I20260621 07:56:06.021855 1130 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:06.021951 1130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:06.022181 1130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:06.024412 1130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "cb3fc6affca046b6a0daf1bf7b864c72"
format_stamp: "Formatted at 2026-06-21 07:56:06 on dist-test-slave-lbpt"
server_key: "1529344e58d070bba523fd1e96e3ca85"
server_key_iv: "3099836dba7f84eb5068201b1ce491e6"
server_key_version: "encryptionkey@0"
I20260621 07:56:06.025023 1130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "cb3fc6affca046b6a0daf1bf7b864c72"
format_stamp: "Formatted at 2026-06-21 07:56:06 on dist-test-slave-lbpt"
server_key: "1529344e58d070bba523fd1e96e3ca85"
server_key_iv: "3099836dba7f84eb5068201b1ce491e6"
server_key_version: "encryptionkey@0"
I20260621 07:56:06.029003 1130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:56:06.031702 1145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:06.033082 1130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:56:06.033234 1130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "cb3fc6affca046b6a0daf1bf7b864c72"
format_stamp: "Formatted at 2026-06-21 07:56:06 on dist-test-slave-lbpt"
server_key: "1529344e58d070bba523fd1e96e3ca85"
server_key_iv: "3099836dba7f84eb5068201b1ce491e6"
server_key_version: "encryptionkey@0"
I20260621 07:56:06.033360 1130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:06.057595 1130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:06.061213 1130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:06.061573 1130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:06.068994 1130 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37743
I20260621 07:56:06.069022 1197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37743 every 8 connection(s)
I20260621 07:56:06.070035 1130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:56:06.070397 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 1130
I20260621 07:56:06.070596 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:56:06.070943 12336 external_mini_cluster.cc:1428] Setting key 3f031e6472fa5a918f09d734bcc9e0af
I20260621 07:56:06.073546 1198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:06.080331 1198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72: Bootstrap starting.
Jun 21 07:56:06 dist-test-slave-lbpt krb5kdc[1117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 07:56:06.083248 1198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:06.084159 1198 log.cc:824] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:06.086848 1198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72: No bootstrap required, opened a new log
I20260621 07:56:06.087618 1201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:06.072501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:33363 (local address 127.0.0.1:37743)
0621 07:56:06.072967 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:06.072987 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:06.073031 (+ 44us) server_negotiation.cc:407] Connection header received
0621 07:56:06.073897 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:06.073921 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:06.074362 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:06.074758 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:06.075498 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:06.077061 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:06.077975 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:06.078706 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:06.078873 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:06.081799 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:06.081806 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:06.081823 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:06.081869 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:06.083989 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:06.084627 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:06.084636 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:06.084645 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:06.084786 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:06.085265 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:06.085269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:06.085271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:06.085819 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:06.086021 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:06.086321 (+ 300us) server_negotiation.cc:299] Negotiation successful
0621 07:56:06.086585 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":130,"threads_started":1}
I20260621 07:56:06.091001 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:56:06.091344 1198 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3fc6affca046b6a0daf1bf7b864c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } }
I20260621 07:56:06.091598 1198 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:56:06.091675 1198 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3fc6affca046b6a0daf1bf7b864c72, State: Initialized, Role: FOLLOWER
I20260621 07:56:06.092093 1198 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3fc6affca046b6a0daf1bf7b864c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } }
I20260621 07:56:06.092247 1198 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:56:06.092319 1198 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:56:06.092418 1198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:06.093591 1198 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3fc6affca046b6a0daf1bf7b864c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } }
I20260621 07:56:06.093954 1198 leader_election.cc:302] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb3fc6affca046b6a0daf1bf7b864c72; no voters:
I20260621 07:56:06.094300 1198 leader_election.cc:288] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:56:06.094453 1208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:56:06.094740 1208 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [term 1 LEADER]: Becoming Leader. State: Replica: cb3fc6affca046b6a0daf1bf7b864c72, State: Running, Role: LEADER
I20260621 07:56:06.095158 1208 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3fc6affca046b6a0daf1bf7b864c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } }
I20260621 07:56:06.095691 1198 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:06.096357 1208 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb3fc6affca046b6a0daf1bf7b864c72. Latest consensus state: current_term: 1 leader_uuid: "cb3fc6affca046b6a0daf1bf7b864c72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3fc6affca046b6a0daf1bf7b864c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } } }
I20260621 07:56:06.096472 1208 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:06.097095 1209 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb3fc6affca046b6a0daf1bf7b864c72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3fc6affca046b6a0daf1bf7b864c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } } }
I20260621 07:56:06.097201 1209 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:06.098615 1215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:56:06.102102 1215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:56:06.103868 1201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:06.093218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:39435 (local address 127.0.0.1:37743)
0621 07:56:06.093342 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:06.093353 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:06.093491 (+ 138us) server_negotiation.cc:407] Connection header received
0621 07:56:06.093629 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:06.093631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:06.093671 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:06.093761 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:06.094584 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:06.095426 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:06.097353 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:06.097715 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:06.097814 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:06.099851 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:06.099856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:06.099863 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:06.099906 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:06.101653 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:06.102286 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:06.102290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:06.102292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:06.102388 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:06.102811 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:06.102815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:06.102817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:06.103053 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:06.103197 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:06.103586 (+ 389us) server_negotiation.cc:299] Negotiation successful
0621 07:56:06.103701 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:56:06.108115 1215 catalog_manager.cc:1347] Generated new cluster ID: 2dffd185c65444ec9048e406b5660001
I20260621 07:56:06.108196 1215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:56:06.123574 1215 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:56:06.124894 1215 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:56:06.137830 1215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb3fc6affca046b6a0daf1bf7b864c72: Generated new TSK 0
I20260621 07:56:06.138792 1215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:56:06.144073 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 1130
W20260621 07:56:06.149626 1115 connection.cc:535] client connection to 127.0.0.1:37743 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37743): Connection reset by peer (error 104)
2026-06-21T07:56:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:56:06 dist-test-slave-lbpt krb5kdc[1229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:56:06 dist-test-slave-lbpt krb5kdc[1229](info): set up 2 sockets
Jun 21 07:56:06 dist-test-slave-lbpt krb5kdc[1229](info): commencing operation
krb5kdc: starting...
W20260621 07:56:07.190840 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:56:07 dist-test-slave-lbpt krb5kdc[1229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:56:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:56:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:56:07.317008 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:07.428691 1242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:07.428982 1242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:07.429024 1242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:07.433102 1242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:07.433171 1242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:07.433189 1242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:07.433205 1242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:07.433221 1242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:07.438779 1242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:07.439800 1242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:07.440785 1242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:07.449414 1248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:07.449440 1250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:07.449743 1242 server_base.cc:1034] running on GCE node
W20260621 07:56:07.449800 1247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:07.450532 1242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:07.451691 1242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:07.452881 1242 hybrid_clock.cc:648] HybridClock initialized: now 1782028567452861 us; error 38 us; skew 500 ppm
Jun 21 07:56:07 dist-test-slave-lbpt krb5kdc[1229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:07.459940 1242 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:56:07.460935 1242 webserver.cc:460] Webserver started at http://127.0.0.1:37339/ using document root <none> and password file <none>
I20260621 07:56:07.461515 1242 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:07.461604 1242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:07.461838 1242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:07.464031 1242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "06f4314d286b47af90762e76e736f018"
format_stamp: "Formatted at 2026-06-21 07:56:07 on dist-test-slave-lbpt"
server_key: "16f3c05f569e2fc17620efc0f2744a56"
server_key_iv: "865b3629f48bdeffc213cc98029f802b"
server_key_version: "encryptionkey@0"
I20260621 07:56:07.464632 1242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "06f4314d286b47af90762e76e736f018"
format_stamp: "Formatted at 2026-06-21 07:56:07 on dist-test-slave-lbpt"
server_key: "16f3c05f569e2fc17620efc0f2744a56"
server_key_iv: "865b3629f48bdeffc213cc98029f802b"
server_key_version: "encryptionkey@0"
I20260621 07:56:07.468457 1242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:56:07.471025 1257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:07.472255 1242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:56:07.472396 1242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "06f4314d286b47af90762e76e736f018"
format_stamp: "Formatted at 2026-06-21 07:56:07 on dist-test-slave-lbpt"
server_key: "16f3c05f569e2fc17620efc0f2744a56"
server_key_iv: "865b3629f48bdeffc213cc98029f802b"
server_key_version: "encryptionkey@0"
I20260621 07:56:07.472515 1242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:07.492512 1242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:07.495630 1242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:07.495850 1242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:07.503094 1242 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34665
I20260621 07:56:07.503116 1309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34665 every 8 connection(s)
I20260621 07:56:07.504050 1242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:56:07.506946 1310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:07.513274 1310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018: Bootstrap starting.
I20260621 07:56:07.513942 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 1242
I20260621 07:56:07.514077 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:56:07.514350 12336 external_mini_cluster.cc:1428] Setting key 3cd9ea757cb405eb5c0ac5ead85e607c
I20260621 07:56:07.516259 1310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:07.517164 1310 log.cc:824] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:07.519479 1310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018: No bootstrap required, opened a new log
I20260621 07:56:07.523952 1310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f4314d286b47af90762e76e736f018" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34665 } }
I20260621 07:56:07.524235 1310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:56:07.524281 1310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f4314d286b47af90762e76e736f018, State: Initialized, Role: FOLLOWER
I20260621 07:56:07.524748 1313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:07.515759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:56929 (local address 127.0.0.1:34665)
0621 07:56:07.516222 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:07.516242 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:07.516286 (+ 44us) server_negotiation.cc:407] Connection header received
0621 07:56:07.517185 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:07.517212 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:07.517661 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:07.518007 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:07.518453 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:07.519873 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:07.520782 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:07.521519 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:07.521676 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:07.522660 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:07.522667 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:07.522684 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:07.523210 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:07.523276 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:07.523307 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:07.523443 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:07.523632 (+ 189us) server_negotiation.cc:299] Negotiation successful
0621 07:56:07.523770 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":128,"threads_started":1}
I20260621 07:56:07.524768 1310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f4314d286b47af90762e76e736f018" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34665 } }
I20260621 07:56:07.524914 1310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:56:07.524984 1310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:56:07.525105 1310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:07.526134 1310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f4314d286b47af90762e76e736f018" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34665 } }
I20260621 07:56:07.526521 1310 leader_election.cc:302] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06f4314d286b47af90762e76e736f018; no voters:
I20260621 07:56:07.526882 1310 leader_election.cc:288] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:56:07.526947 1315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:56:07.527153 1315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [term 1 LEADER]: Becoming Leader. State: Replica: 06f4314d286b47af90762e76e736f018, State: Running, Role: LEADER
I20260621 07:56:07.527489 1315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f4314d286b47af90762e76e736f018" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34665 } }
I20260621 07:56:07.527946 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:56:07.528187 1310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:07.529376 1317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06f4314d286b47af90762e76e736f018. Latest consensus state: current_term: 1 leader_uuid: "06f4314d286b47af90762e76e736f018" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f4314d286b47af90762e76e736f018" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34665 } } }
I20260621 07:56:07.529378 1316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06f4314d286b47af90762e76e736f018" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f4314d286b47af90762e76e736f018" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34665 } } }
I20260621 07:56:07.529536 1317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:07.529547 1316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:07.529969 1328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:56:07.533160 1328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:56:07.535892 1313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:07.530371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:45163 (local address 127.0.0.1:34665)
0621 07:56:07.530499 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:07.530503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:07.531019 (+ 516us) server_negotiation.cc:407] Connection header received
0621 07:56:07.531302 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:07.531306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:07.531358 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:07.531410 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:07.532027 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:07.532835 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:07.534022 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:07.534317 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:07.534459 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:07.535297 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:07.535301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:07.535303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:07.535317 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:07.535345 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:07.535359 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:07.535452 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:07.535651 (+ 199us) server_negotiation.cc:299] Negotiation successful
0621 07:56:07.535687 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:56:07.539187 1328 catalog_manager.cc:1347] Generated new cluster ID: 73ab1dc6d3df4fceab43ab5e13a3f1a1
I20260621 07:56:07.539255 1328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:56:07.569324 1328 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:56:07.570391 1328 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:56:07.578397 1328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06f4314d286b47af90762e76e736f018: Generated new TSK 0
I20260621 07:56:07.579114 1328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:56:07.589700 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 1242
W20260621 07:56:07.595150 1227 connection.cc:535] client connection to 127.0.0.1:34665 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34665): Connection reset by peer (error 104)
2026-06-21T07:56:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:56:07 dist-test-slave-lbpt krb5kdc[1341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:56:07 dist-test-slave-lbpt krb5kdc[1341](info): set up 2 sockets
Jun 21 07:56:07 dist-test-slave-lbpt krb5kdc[1341](info): commencing operation
krb5kdc: starting...
W20260621 07:56:08.642765 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:56:08 dist-test-slave-lbpt krb5kdc[1341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:56:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:56:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:56:08.771656 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:08.885038 1354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:08.885377 1354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:08.885461 1354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:08.889569 1354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:08.889680 1354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:08.889719 1354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:08.889755 1354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:08.889796 1354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:08.895334 1354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35449
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:08.896562 1354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:08.897663 1354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:08.906688 1359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:08.906694 1362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:08.906694 1360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:08.906948 1354 server_base.cc:1034] running on GCE node
I20260621 07:56:08.907686 1354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:08.908957 1354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:08.910135 1354 hybrid_clock.cc:648] HybridClock initialized: now 1782028568910126 us; error 37 us; skew 500 ppm
Jun 21 07:56:08 dist-test-slave-lbpt krb5kdc[1341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:08.913205 1354 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:56:08.914381 1354 webserver.cc:460] Webserver started at http://127.0.0.1:39433/ using document root <none> and password file <none>
I20260621 07:56:08.915091 1354 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:08.915162 1354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:08.915391 1354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:08.917588 1354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "9b016980049044268c42a21ce75fc5ee"
format_stamp: "Formatted at 2026-06-21 07:56:08 on dist-test-slave-lbpt"
server_key: "b76c58fd5861b8dcc102f90397d0fb82"
server_key_iv: "538a861a3bce4b34f32fa727e6b3db0d"
server_key_version: "encryptionkey@0"
I20260621 07:56:08.918203 1354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "9b016980049044268c42a21ce75fc5ee"
format_stamp: "Formatted at 2026-06-21 07:56:08 on dist-test-slave-lbpt"
server_key: "b76c58fd5861b8dcc102f90397d0fb82"
server_key_iv: "538a861a3bce4b34f32fa727e6b3db0d"
server_key_version: "encryptionkey@0"
I20260621 07:56:08.922287 1354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 07:56:08.925078 1369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:08.926374 1354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:56:08.926589 1354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "9b016980049044268c42a21ce75fc5ee"
format_stamp: "Formatted at 2026-06-21 07:56:08 on dist-test-slave-lbpt"
server_key: "b76c58fd5861b8dcc102f90397d0fb82"
server_key_iv: "538a861a3bce4b34f32fa727e6b3db0d"
server_key_version: "encryptionkey@0"
I20260621 07:56:08.926764 1354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:08.931664 1354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:08.935307 1354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:08.935521 1354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:08.943295 1354 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35449
I20260621 07:56:08.943320 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35449 every 8 connection(s)
I20260621 07:56:08.944371 1354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:56:08.947618 1422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:08.948402 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 1354
I20260621 07:56:08.948550 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:56:08.948897 12336 external_mini_cluster.cc:1428] Setting key 9d4672d7724b92f6eb28d329bdfad1a8
I20260621 07:56:08.954363 1422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee: Bootstrap starting.
I20260621 07:56:08.954725 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:08.950498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:39245 (local address 127.0.0.1:35449)
0621 07:56:08.951033 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:08.951049 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:08.951087 (+ 38us) server_negotiation.cc:407] Connection header received
0621 07:56:08.951736 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:08.951753 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:08.952105 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:08.952576 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:08.953379 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:08.953383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:08.953416 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:08.953427 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:08.953485 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:08.953505 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:08.953657 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:08.953735 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 07:56:08.953896 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":170,"threads_started":1}
I20260621 07:56:08.956970 1422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:08.957603 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:56:08.957818 1422 log.cc:824] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:08.960086 1422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee: No bootstrap required, opened a new log
I20260621 07:56:08.961906 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:08.959864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:39835 (local address 127.0.0.1:35449)
0621 07:56:08.959996 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:08.960001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:08.960225 (+ 224us) server_negotiation.cc:407] Connection header received
0621 07:56:08.960487 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:08.960489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:08.960524 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:08.960601 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:08.961355 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:08.961359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:08.961361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:08.961372 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:08.961398 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:08.961409 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:08.961486 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:08.961579 (+ 93us) server_negotiation.cc:299] Negotiation successful
0621 07:56:08.961616 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:56:08.964372 1422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b016980049044268c42a21ce75fc5ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35449 } }
I20260621 07:56:08.964638 1422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:56:08.964712 1422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b016980049044268c42a21ce75fc5ee, State: Initialized, Role: FOLLOWER
I20260621 07:56:08.965204 1422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b016980049044268c42a21ce75fc5ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35449 } }
I20260621 07:56:08.965364 1422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:56:08.965447 1422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:56:08.965549 1422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:08.966615 1422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b016980049044268c42a21ce75fc5ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35449 } }
I20260621 07:56:08.966985 1422 leader_election.cc:302] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b016980049044268c42a21ce75fc5ee; no voters:
I20260621 07:56:08.967370 1422 leader_election.cc:288] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:56:08.967461 1432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:56:08.967674 1432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [term 1 LEADER]: Becoming Leader. State: Replica: 9b016980049044268c42a21ce75fc5ee, State: Running, Role: LEADER
I20260621 07:56:08.968060 1432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b016980049044268c42a21ce75fc5ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35449 } }
I20260621 07:56:08.968830 1422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:08.969954 1433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b016980049044268c42a21ce75fc5ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b016980049044268c42a21ce75fc5ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35449 } } }
I20260621 07:56:08.970104 1433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:08.970407 1434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b016980049044268c42a21ce75fc5ee. Latest consensus state: current_term: 1 leader_uuid: "9b016980049044268c42a21ce75fc5ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b016980049044268c42a21ce75fc5ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35449 } } }
I20260621 07:56:08.970539 1434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:08.970556 1441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:56:08.973542 1441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:56:08.979568 1441 catalog_manager.cc:1347] Generated new cluster ID: e980c86088684b63af19b19b90c3b53a
I20260621 07:56:08.979671 1441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:56:08.989387 1441 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:56:08.989761 1441 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:56:08.997390 1441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b016980049044268c42a21ce75fc5ee: Generated new TSK 0
I20260621 07:56:08.998364 1441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:56:09.007948 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 1354
2026-06-21T07:56:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:56:09 dist-test-slave-lbpt krb5kdc[1453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:56:09 dist-test-slave-lbpt krb5kdc[1453](info): set up 2 sockets
Jun 21 07:56:09 dist-test-slave-lbpt krb5kdc[1453](info): commencing operation
krb5kdc: starting...
W20260621 07:56:10.058823 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:56:10 dist-test-slave-lbpt krb5kdc[1453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:56:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:56:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:56:10.186111 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:45215
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:32835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:45215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:10.300163 1466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:10.300446 1466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:10.300490 1466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:10.304581 1466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:10.304646 1466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:10.304665 1466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:10.304682 1466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:10.304697 1466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:10.310123 1466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:32835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:45215
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:45215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:10.311327 1466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:10.312374 1466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:10.320778 1471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:10.320844 1474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:10.320860 1472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:10.321019 1466 server_base.cc:1034] running on GCE node
I20260621 07:56:10.321717 1466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:10.323378 1466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:10.324559 1466 hybrid_clock.cc:648] HybridClock initialized: now 1782028570324519 us; error 50 us; skew 500 ppm
Jun 21 07:56:10 dist-test-slave-lbpt krb5kdc[1453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028570, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:10.328166 1466 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:56:10.329137 1466 webserver.cc:460] Webserver started at http://127.12.12.62:32893/ using document root <none> and password file <none>
I20260621 07:56:10.329711 1466 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:10.329799 1466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:10.330026 1466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:10.332945 1466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "c7a4ea4968fd491a8ca93f3f7a9e1a01"
format_stamp: "Formatted at 2026-06-21 07:56:10 on dist-test-slave-lbpt"
server_key: "eee219ecf9d8cfc194241641918e188a"
server_key_iv: "991fae90caa30c91f48cd26db7fbb444"
server_key_version: "encryptionkey@0"
I20260621 07:56:10.333689 1466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "c7a4ea4968fd491a8ca93f3f7a9e1a01"
format_stamp: "Formatted at 2026-06-21 07:56:10 on dist-test-slave-lbpt"
server_key: "eee219ecf9d8cfc194241641918e188a"
server_key_iv: "991fae90caa30c91f48cd26db7fbb444"
server_key_version: "encryptionkey@0"
I20260621 07:56:10.339154 1466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 07:56:10.342394 1481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:10.344122 1466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:56:10.344267 1466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "c7a4ea4968fd491a8ca93f3f7a9e1a01"
format_stamp: "Formatted at 2026-06-21 07:56:10 on dist-test-slave-lbpt"
server_key: "eee219ecf9d8cfc194241641918e188a"
server_key_iv: "991fae90caa30c91f48cd26db7fbb444"
server_key_version: "encryptionkey@0"
I20260621 07:56:10.344421 1466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:10.371073 1466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:10.374259 1466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:10.374523 1466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:10.381709 1466 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:45215
I20260621 07:56:10.381775 1533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:45215 every 8 connection(s)
I20260621 07:56:10.382777 1466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:56:10.383229 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 1466
I20260621 07:56:10.383399 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:56:10.383726 12336 external_mini_cluster.cc:1428] Setting key c4c833c6d3f2e5ebbe0e3c6bbba432a0
I20260621 07:56:10.386327 1534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:10.393710 1534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01: Bootstrap starting.
Jun 21 07:56:10 dist-test-slave-lbpt krb5kdc[1453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:56:10.396626 1534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:10.397563 1534 log.cc:824] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:10.400188 1534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01: No bootstrap required, opened a new log
I20260621 07:56:10.400790 1537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:10.385247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37063 (local address 127.12.12.62:45215)
0621 07:56:10.385797 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:10.385816 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:10.385861 (+ 45us) server_negotiation.cc:407] Connection header received
0621 07:56:10.386782 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:10.386806 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:10.387256 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:10.387787 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:10.388535 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:10.390140 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:10.391332 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:10.392038 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:10.392214 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:10.395168 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:10.395175 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:10.395192 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:10.395228 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:10.397246 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:10.397850 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:10.397857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:10.397864 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:10.398033 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:10.398394 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:10.398396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:10.398398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:10.398981 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:10.399187 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:10.399495 (+ 308us) server_negotiation.cc:299] Negotiation successful
0621 07:56:10.399756 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":182,"threads_started":1}
I20260621 07:56:10.403890 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:56:10.404579 1534 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a4ea4968fd491a8ca93f3f7a9e1a01" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 45215 } }
I20260621 07:56:10.404811 1534 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:56:10.404881 1534 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a4ea4968fd491a8ca93f3f7a9e1a01, State: Initialized, Role: FOLLOWER
I20260621 07:56:10.405386 1534 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a4ea4968fd491a8ca93f3f7a9e1a01" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 45215 } }
I20260621 07:56:10.405519 1534 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:56:10.405596 1534 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:56:10.405709 1534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:10.407021 1534 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a4ea4968fd491a8ca93f3f7a9e1a01" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 45215 } }
I20260621 07:56:10.407388 1534 leader_election.cc:302] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7a4ea4968fd491a8ca93f3f7a9e1a01; no voters:
I20260621 07:56:10.407733 1534 leader_election.cc:288] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:56:10.407883 1544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:56:10.408141 1544 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [term 1 LEADER]: Becoming Leader. State: Replica: c7a4ea4968fd491a8ca93f3f7a9e1a01, State: Running, Role: LEADER
I20260621 07:56:10.408466 1544 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a4ea4968fd491a8ca93f3f7a9e1a01" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 45215 } }
I20260621 07:56:10.409063 1534 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:10.409587 1544 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7a4ea4968fd491a8ca93f3f7a9e1a01. Latest consensus state: current_term: 1 leader_uuid: "c7a4ea4968fd491a8ca93f3f7a9e1a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a4ea4968fd491a8ca93f3f7a9e1a01" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 45215 } } }
I20260621 07:56:10.409706 1544 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:10.410029 1546 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7a4ea4968fd491a8ca93f3f7a9e1a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a4ea4968fd491a8ca93f3f7a9e1a01" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 45215 } } }
I20260621 07:56:10.410117 1546 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01 [sys.catalog]: This master's current role is: LEADER
W20260621 07:56:10.414963 1559 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:56:10.415050 1559 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:56:10.415133 1550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:56:10.416051 1537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:10.406179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54081 (local address 127.12.12.62:45215)
0621 07:56:10.406337 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:10.406342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:10.406361 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:56:10.406541 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:10.406544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:10.406589 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:10.406662 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:10.407398 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:10.408186 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:10.409315 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:10.409640 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:10.410477 (+ 837us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:10.411786 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:10.411789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:10.411792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:10.411817 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:10.413611 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:10.414223 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:10.414226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:10.414228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:10.414325 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:10.414763 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:10.414766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:10.414768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:10.414977 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:10.415716 (+ 739us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:10.415812 (+ 96us) server_negotiation.cc:299] Negotiation successful
0621 07:56:10.415907 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 07:56:10.416059 1550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:56:10.421438 1550 catalog_manager.cc:1347] Generated new cluster ID: db7d1127f9734abc82ef08c8c22429a3
I20260621 07:56:10.421509 1550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:56:10.429448 1550 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:56:10.430701 1550 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:56:10.443917 1550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7a4ea4968fd491a8ca93f3f7a9e1a01: Generated new TSK 0
I20260621 07:56:10.444677 1550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:56:10.455260 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 1466
W20260621 07:56:10.460613 1451 connection.cc:535] client connection to 127.12.12.62:45215 recv error: Network error: failed to read from TLS socket (remote: 127.12.12.62:45215): Connection reset by peer (error 104)
2026-06-21T07:56:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:56:10 dist-test-slave-lbpt krb5kdc[1565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:56:10 dist-test-slave-lbpt krb5kdc[1565](info): set up 2 sockets
Jun 21 07:56:10 dist-test-slave-lbpt krb5kdc[1565](info): commencing operation
krb5kdc: starting...
W20260621 07:56:11.506767 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:56:11 dist-test-slave-lbpt krb5kdc[1565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:56:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:56:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:56:11.634575 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:34749
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:34749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:11.746184 1578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:11.746544 1578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:11.746629 1578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:11.750701 1578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:11.750801 1578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:11.750840 1578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:11.750893 1578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:11.750929 1578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:11.756335 1578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:34749
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:34749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:11.757431 1578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:11.758520 1578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:11.767562 1584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:11.767567 1583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:11.768038 1586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:11.770241 1578 server_base.cc:1034] running on GCE node
I20260621 07:56:11.770934 1578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:11.772123 1578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:11.773317 1578 hybrid_clock.cc:648] HybridClock initialized: now 1782028571773284 us; error 54 us; skew 500 ppm
Jun 21 07:56:11 dist-test-slave-lbpt krb5kdc[1565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028571, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:11.776248 1578 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:56:11.777262 1578 webserver.cc:460] Webserver started at http://127.12.12.62:33061/ using document root <none> and password file <none>
I20260621 07:56:11.777854 1578 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:11.777949 1578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:11.778240 1578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:11.780532 1578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "fe721fb124c84b7ba864c812b02a84cb"
format_stamp: "Formatted at 2026-06-21 07:56:11 on dist-test-slave-lbpt"
server_key: "fec5ad4fd654235abfb8a92cadf15c09"
server_key_iv: "8fe15f3e71e7bf6e66d8907488964575"
server_key_version: "encryptionkey@0"
I20260621 07:56:11.781134 1578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "fe721fb124c84b7ba864c812b02a84cb"
format_stamp: "Formatted at 2026-06-21 07:56:11 on dist-test-slave-lbpt"
server_key: "fec5ad4fd654235abfb8a92cadf15c09"
server_key_iv: "8fe15f3e71e7bf6e66d8907488964575"
server_key_version: "encryptionkey@0"
I20260621 07:56:11.785378 1578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 07:56:11.788242 1593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:11.789851 1578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:56:11.790027 1578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "fe721fb124c84b7ba864c812b02a84cb"
format_stamp: "Formatted at 2026-06-21 07:56:11 on dist-test-slave-lbpt"
server_key: "fec5ad4fd654235abfb8a92cadf15c09"
server_key_iv: "8fe15f3e71e7bf6e66d8907488964575"
server_key_version: "encryptionkey@0"
I20260621 07:56:11.790164 1578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:11.801872 1578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:11.804888 1578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:11.805111 1578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:11.812536 1578 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:34749
I20260621 07:56:11.812553 1645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:34749 every 8 connection(s)
I20260621 07:56:11.813529 1578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:56:11.816409 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:11.821121 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 1578
I20260621 07:56:11.821281 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:56:11.821573 12336 external_mini_cluster.cc:1428] Setting key d4ef8765fc7e09709592830687db7623
I20260621 07:56:11.822149 1646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb: Bootstrap starting.
I20260621 07:56:11.824579 1646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:11.825467 1646 log.cc:824] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:11.827759 1646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb: No bootstrap required, opened a new log
I20260621 07:56:11.831292 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:11.822989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54383 (local address 127.12.12.62:34749)
0621 07:56:11.823412 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:11.823431 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:11.823467 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:56:11.824134 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:11.824152 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:11.824507 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:11.824836 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:11.825168 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:11.826366 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:11.827681 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:11.828318 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:11.828445 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:11.829437 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:11.829442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:11.829455 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:11.829870 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:11.829926 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:11.829949 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:11.830057 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:11.830266 (+ 209us) server_negotiation.cc:299] Negotiation successful
0621 07:56:11.830453 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":120,"threads_started":1}
I20260621 07:56:11.832077 1646 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe721fb124c84b7ba864c812b02a84cb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34749 } }
I20260621 07:56:11.832325 1646 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:56:11.832403 1646 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe721fb124c84b7ba864c812b02a84cb, State: Initialized, Role: FOLLOWER
I20260621 07:56:11.832868 1646 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe721fb124c84b7ba864c812b02a84cb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34749 } }
I20260621 07:56:11.833017 1646 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:56:11.833086 1646 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:56:11.833189 1646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:11.834272 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:56:11.834295 1646 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe721fb124c84b7ba864c812b02a84cb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34749 } }
I20260621 07:56:11.834666 1646 leader_election.cc:302] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe721fb124c84b7ba864c812b02a84cb; no voters:
I20260621 07:56:11.835012 1646 leader_election.cc:288] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:56:11.835110 1651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:56:11.835306 1651 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [term 1 LEADER]: Becoming Leader. State: Replica: fe721fb124c84b7ba864c812b02a84cb, State: Running, Role: LEADER
I20260621 07:56:11.835603 1651 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe721fb124c84b7ba864c812b02a84cb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34749 } }
I20260621 07:56:11.836406 1646 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:11.836809 1653 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe721fb124c84b7ba864c812b02a84cb. Latest consensus state: current_term: 1 leader_uuid: "fe721fb124c84b7ba864c812b02a84cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe721fb124c84b7ba864c812b02a84cb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34749 } } }
I20260621 07:56:11.836959 1653 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:11.837273 1652 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe721fb124c84b7ba864c812b02a84cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe721fb124c84b7ba864c812b02a84cb" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34749 } } }
I20260621 07:56:11.837364 1652 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:11.837824 1661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:56:11.841171 1661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:56:11.842383 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:11.836437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52499 (local address 127.12.12.62:34749)
0621 07:56:11.836549 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:11.836553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:11.837457 (+ 904us) server_negotiation.cc:407] Connection header received
0621 07:56:11.837631 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:11.837634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:11.837687 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:11.837761 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:11.838169 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:11.838951 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:11.840085 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:11.840404 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:11.840477 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:11.841660 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:11.841663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:11.841667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:11.841681 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:11.841707 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:11.841721 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:11.841788 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:11.842207 (+ 419us) server_negotiation.cc:299] Negotiation successful
0621 07:56:11.842248 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:56:11.847159 1661 catalog_manager.cc:1347] Generated new cluster ID: 471c4d3a46284a759b758d0fc8eb522c
I20260621 07:56:11.847232 1661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:56:11.875834 1661 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:56:11.876883 1661 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:56:11.886917 1661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe721fb124c84b7ba864c812b02a84cb: Generated new TSK 0
I20260621 07:56:11.887686 1661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:56:11.897397 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 1578
W20260621 07:56:11.902915 1563 connection.cc:535] client connection to 127.12.12.62:34749 recv error: Network error: failed to read from TLS socket (remote: 127.12.12.62:34749): Connection reset by peer (error 104)
2026-06-21T07:56:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:56:11 dist-test-slave-lbpt krb5kdc[1677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:56:11 dist-test-slave-lbpt krb5kdc[1677](info): set up 2 sockets
Jun 21 07:56:11 dist-test-slave-lbpt krb5kdc[1677](info): commencing operation
krb5kdc: starting...
W20260621 07:56:12.942759 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:56:13 dist-test-slave-lbpt krb5kdc[1677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:56:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:56:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:56:13.065122 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:32857
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:32857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:13.176851 1690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:13.177120 1690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:13.177162 1690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:13.181151 1690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:13.181216 1690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:13.181236 1690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:13.181252 1690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:13.181267 1690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:13.186765 1690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:32857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:32857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:13.187892 1690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:13.188898 1690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:13.197618 1698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:13.197659 1695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:13.197628 1696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:13.198058 1690 server_base.cc:1034] running on GCE node
I20260621 07:56:13.198640 1690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:13.199705 1690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:13.200899 1690 hybrid_clock.cc:648] HybridClock initialized: now 1782028573200884 us; error 26 us; skew 500 ppm
Jun 21 07:56:13 dist-test-slave-lbpt krb5kdc[1677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028573, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:13.208597 1690 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:56:13.209548 1690 webserver.cc:460] Webserver started at http://127.12.12.62:43689/ using document root <none> and password file <none>
I20260621 07:56:13.210094 1690 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:13.210156 1690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:13.210332 1690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:13.212539 1690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "2028cb04284b48e9b6d316b63661b7c8"
format_stamp: "Formatted at 2026-06-21 07:56:13 on dist-test-slave-lbpt"
server_key: "beac8d1df4b33401466f8ef60f80627a"
server_key_iv: "f323bb821197880751a55b2c3f49f3cd"
server_key_version: "encryptionkey@0"
I20260621 07:56:13.213081 1690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "2028cb04284b48e9b6d316b63661b7c8"
format_stamp: "Formatted at 2026-06-21 07:56:13 on dist-test-slave-lbpt"
server_key: "beac8d1df4b33401466f8ef60f80627a"
server_key_iv: "f323bb821197880751a55b2c3f49f3cd"
server_key_version: "encryptionkey@0"
I20260621 07:56:13.217068 1690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:56:13.219580 1705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:13.220873 1690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:56:13.221022 1690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "2028cb04284b48e9b6d316b63661b7c8"
format_stamp: "Formatted at 2026-06-21 07:56:13 on dist-test-slave-lbpt"
server_key: "beac8d1df4b33401466f8ef60f80627a"
server_key_iv: "f323bb821197880751a55b2c3f49f3cd"
server_key_version: "encryptionkey@0"
I20260621 07:56:13.221129 1690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:13.225845 1690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:13.229162 1690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:13.229364 1690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:13.236528 1690 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:32857
I20260621 07:56:13.236557 1757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:32857 every 8 connection(s)
I20260621 07:56:13.237504 1690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:56:13.240374 1758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:13.241281 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 1690
I20260621 07:56:13.241433 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:56:13.241750 12336 external_mini_cluster.cc:1428] Setting key 9486a737de991e2b6c45a4dc25aa4850
I20260621 07:56:13.246873 1758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8: Bootstrap starting.
I20260621 07:56:13.247161 1761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:13.243227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32999 (local address 127.12.12.62:32857)
0621 07:56:13.243696 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:13.243710 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:13.243746 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:56:13.244418 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:13.244435 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:13.244718 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:13.245065 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:13.245869 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:13.245873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:13.245885 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:13.245896 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:13.245949 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:13.245967 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:13.246115 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:13.246187 (+ 72us) server_negotiation.cc:299] Negotiation successful
0621 07:56:13.246349 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":132,"threads_started":1}
I20260621 07:56:13.249143 1758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:13.249862 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:56:13.250016 1758 log.cc:824] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:13.252079 1758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8: No bootstrap required, opened a new log
I20260621 07:56:13.254109 1761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:13.252236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39305 (local address 127.12.12.62:32857)
0621 07:56:13.252409 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:13.252414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:13.252717 (+ 303us) server_negotiation.cc:407] Connection header received
0621 07:56:13.252833 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:13.252835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:13.252869 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:13.252949 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:13.253668 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:13.253671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:13.253673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:13.253684 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:13.253714 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:13.253729 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:13.253808 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:13.253901 (+ 93us) server_negotiation.cc:299] Negotiation successful
0621 07:56:13.253938 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:56:13.256471 1758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2028cb04284b48e9b6d316b63661b7c8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 32857 } }
I20260621 07:56:13.256717 1758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:56:13.256776 1758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2028cb04284b48e9b6d316b63661b7c8, State: Initialized, Role: FOLLOWER
I20260621 07:56:13.257189 1758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2028cb04284b48e9b6d316b63661b7c8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 32857 } }
I20260621 07:56:13.257332 1758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:56:13.257397 1758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:56:13.257498 1758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:13.258507 1758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2028cb04284b48e9b6d316b63661b7c8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 32857 } }
I20260621 07:56:13.258880 1758 leader_election.cc:302] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2028cb04284b48e9b6d316b63661b7c8; no voters:
I20260621 07:56:13.259197 1758 leader_election.cc:288] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:56:13.259326 1768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:56:13.259567 1768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [term 1 LEADER]: Becoming Leader. State: Replica: 2028cb04284b48e9b6d316b63661b7c8, State: Running, Role: LEADER
I20260621 07:56:13.259948 1768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2028cb04284b48e9b6d316b63661b7c8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 32857 } }
I20260621 07:56:13.260542 1758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:13.261626 1770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2028cb04284b48e9b6d316b63661b7c8. Latest consensus state: current_term: 1 leader_uuid: "2028cb04284b48e9b6d316b63661b7c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2028cb04284b48e9b6d316b63661b7c8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 32857 } } }
I20260621 07:56:13.261760 1770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:13.262046 1769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2028cb04284b48e9b6d316b63661b7c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2028cb04284b48e9b6d316b63661b7c8" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 32857 } } }
I20260621 07:56:13.262143 1769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:13.262290 1777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:56:13.265285 1777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:56:13.270951 1777 catalog_manager.cc:1347] Generated new cluster ID: 25bd045fb3aa4d24925cfb625196320f
I20260621 07:56:13.271018 1777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:56:13.282284 1777 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:56:13.282657 1777 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:56:13.294862 1777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2028cb04284b48e9b6d316b63661b7c8: Generated new TSK 0
I20260621 07:56:13.295601 1777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:56:13.308123 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 1690
2026-06-21T07:56:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:56:13 dist-test-slave-lbpt krb5kdc[1789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:56:13 dist-test-slave-lbpt krb5kdc[1789](info): set up 2 sockets
Jun 21 07:56:13 dist-test-slave-lbpt krb5kdc[1789](info): commencing operation
krb5kdc: starting...
W20260621 07:56:14.354780 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:56:14 dist-test-slave-lbpt krb5kdc[1789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:56:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:56:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:56:14.474052 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:44313
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:44313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:14.584932 1802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:14.585201 1802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:14.585242 1802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:14.589233 1802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:14.589346 1802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:14.589386 1802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:14.589442 1802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:14.589479 1802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:14.594837 1802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:44313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:44313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:14.595969 1802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:14.597025 1802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:14.605553 1808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:14.605556 1810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:14.605626 1807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:14.606079 1802 server_base.cc:1034] running on GCE node
I20260621 07:56:14.606652 1802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:14.607749 1802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:14.608942 1802 hybrid_clock.cc:648] HybridClock initialized: now 1782028574608912 us; error 45 us; skew 500 ppm
Jun 21 07:56:14 dist-test-slave-lbpt krb5kdc[1789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028574, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:14.611702 1802 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:56:14.612537 1802 webserver.cc:460] Webserver started at http://127.12.12.62:34353/ using document root <none> and password file <none>
I20260621 07:56:14.613052 1802 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:14.613109 1802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:14.613269 1802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:14.615310 1802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "855432a5a1bf4971865561b4643c8710"
format_stamp: "Formatted at 2026-06-21 07:56:14 on dist-test-slave-lbpt"
server_key: "c4d17c269f7eb9f040afca4ffbf89372"
server_key_iv: "b9ea9c941f1c6ac6b64df7df979818a6"
server_key_version: "encryptionkey@0"
I20260621 07:56:14.615805 1802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "855432a5a1bf4971865561b4643c8710"
format_stamp: "Formatted at 2026-06-21 07:56:14 on dist-test-slave-lbpt"
server_key: "c4d17c269f7eb9f040afca4ffbf89372"
server_key_iv: "b9ea9c941f1c6ac6b64df7df979818a6"
server_key_version: "encryptionkey@0"
I20260621 07:56:14.619470 1802 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:56:14.621932 1817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:14.623178 1802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:56:14.623311 1802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "855432a5a1bf4971865561b4643c8710"
format_stamp: "Formatted at 2026-06-21 07:56:14 on dist-test-slave-lbpt"
server_key: "c4d17c269f7eb9f040afca4ffbf89372"
server_key_iv: "b9ea9c941f1c6ac6b64df7df979818a6"
server_key_version: "encryptionkey@0"
I20260621 07:56:14.623421 1802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:14.638374 1802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:14.641191 1802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:14.641382 1802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:14.648118 1802 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:44313
I20260621 07:56:14.648180 1869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:44313 every 8 connection(s)
I20260621 07:56:14.649096 1802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:56:14.650324 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 1802
I20260621 07:56:14.650475 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:56:14.650810 12336 external_mini_cluster.cc:1428] Setting key eefb560cb55493da6a85e065d1d2b958
I20260621 07:56:14.652387 1870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:14.658806 1870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710: Bootstrap starting.
Jun 21 07:56:14 dist-test-slave-lbpt krb5kdc[1789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:56:14.661409 1870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:14.662324 1870 log.cc:824] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:14.664215 1870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710: No bootstrap required, opened a new log
I20260621 07:56:14.665231 1873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:14.652136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50271 (local address 127.12.12.62:44313)
0621 07:56:14.652528 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:14.652551 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:14.652585 (+ 34us) server_negotiation.cc:407] Connection header received
0621 07:56:14.653246 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:14.653263 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:14.653592 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:14.653875 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:14.654773 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:14.655785 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:14.656946 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:14.657384 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:14.657494 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:14.660306 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:14.660312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:14.660328 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:14.660367 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:14.662401 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:14.662866 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:14.662876 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:14.662882 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:14.662999 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:14.663283 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:14.663288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:14.663289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:14.663682 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:14.663903 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:14.664172 (+ 269us) server_negotiation.cc:299] Negotiation successful
0621 07:56:14.664506 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":144,"threads_started":1}
I20260621 07:56:14.667809 1870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855432a5a1bf4971865561b4643c8710" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44313 } }
I20260621 07:56:14.668062 1870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:56:14.668126 1870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 855432a5a1bf4971865561b4643c8710, State: Initialized, Role: FOLLOWER
I20260621 07:56:14.668607 1870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855432a5a1bf4971865561b4643c8710" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44313 } }
I20260621 07:56:14.668759 1870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:56:14.668824 1870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:56:14.668795 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:56:14.668944 1870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:14.669868 1870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855432a5a1bf4971865561b4643c8710" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44313 } }
I20260621 07:56:14.670243 1870 leader_election.cc:302] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 855432a5a1bf4971865561b4643c8710; no voters:
I20260621 07:56:14.670580 1870 leader_election.cc:288] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:56:14.670678 1879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:56:14.670923 1879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [term 1 LEADER]: Becoming Leader. State: Replica: 855432a5a1bf4971865561b4643c8710, State: Running, Role: LEADER
I20260621 07:56:14.671343 1879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855432a5a1bf4971865561b4643c8710" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44313 } }
I20260621 07:56:14.671905 1870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:14.672587 1881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 855432a5a1bf4971865561b4643c8710. Latest consensus state: current_term: 1 leader_uuid: "855432a5a1bf4971865561b4643c8710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855432a5a1bf4971865561b4643c8710" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44313 } } }
I20260621 07:56:14.672701 1881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:14.673372 1880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "855432a5a1bf4971865561b4643c8710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855432a5a1bf4971865561b4643c8710" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 44313 } } }
I20260621 07:56:14.673466 1880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710 [sys.catalog]: This master's current role is: LEADER
W20260621 07:56:14.677294 1895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:56:14.677415 1895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:56:14.677796 1887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:56:14.678802 1887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:56:14.681219 1873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:14.671482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40657 (local address 127.12.12.62:44313)
0621 07:56:14.671583 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:14.671588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:14.671602 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:56:14.671745 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:14.671755 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:14.671805 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:14.671855 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:14.672846 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:14.673697 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:14.675296 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:14.675678 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:14.675787 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:14.677745 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:14.677749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:14.677751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:14.677776 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:14.679411 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:14.679893 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:14.679896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:14.679898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:14.679986 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:14.680319 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:14.680322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:14.680324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:14.680561 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:14.680692 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:14.680956 (+ 264us) server_negotiation.cc:299] Negotiation successful
0621 07:56:14.681075 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 07:56:14.684886 1887 catalog_manager.cc:1347] Generated new cluster ID: 372efc8a835d48779438ffe670acc5fe
I20260621 07:56:14.684957 1887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:56:14.701280 1887 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:56:14.702661 1887 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:56:14.709816 1887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 855432a5a1bf4971865561b4643c8710: Generated new TSK 0
I20260621 07:56:14.710680 1887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:56:14.720664 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 1802
2026-06-21T07:56:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:56:14 dist-test-slave-lbpt krb5kdc[1901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:56:14 dist-test-slave-lbpt krb5kdc[1901](info): set up 2 sockets
Jun 21 07:56:14 dist-test-slave-lbpt krb5kdc[1901](info): commencing operation
krb5kdc: starting...
W20260621 07:56:15.766875 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:56:15 dist-test-slave-lbpt krb5kdc[1901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:56:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:56:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:56:15.893712 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:35729
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:35729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:16.017366 1914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:16.017664 1914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:16.017709 1914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:16.021970 1914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:16.022035 1914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:16.022056 1914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:16.022099 1914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:16.022140 1914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:16.027721 1914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:35729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:35729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:16.028864 1914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:16.029932 1914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:16.038648 1920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:16.038689 1922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:16.038828 1914 server_base.cc:1034] running on GCE node
W20260621 07:56:16.038648 1919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:16.039520 1914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:16.040735 1914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:16.041949 1914 hybrid_clock.cc:648] HybridClock initialized: now 1782028576041929 us; error 30 us; skew 500 ppm
Jun 21 07:56:16 dist-test-slave-lbpt krb5kdc[1901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028576, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:16.044864 1914 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:56:16.045778 1914 webserver.cc:460] Webserver started at http://127.12.12.62:33873/ using document root <none> and password file <none>
I20260621 07:56:16.046392 1914 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:16.046519 1914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:16.046797 1914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:16.048869 1914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "fc23972cd54e418cb678d9d19faf463e"
format_stamp: "Formatted at 2026-06-21 07:56:16 on dist-test-slave-lbpt"
server_key: "eb89cbaabeccfd9bd563cb4e660cb7d3"
server_key_iv: "4461688260c4929f4cae327a566c0760"
server_key_version: "encryptionkey@0"
I20260621 07:56:16.049459 1914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "fc23972cd54e418cb678d9d19faf463e"
format_stamp: "Formatted at 2026-06-21 07:56:16 on dist-test-slave-lbpt"
server_key: "eb89cbaabeccfd9bd563cb4e660cb7d3"
server_key_iv: "4461688260c4929f4cae327a566c0760"
server_key_version: "encryptionkey@0"
I20260621 07:56:16.053411 1914 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260621 07:56:16.056159 1929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:16.057691 1914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:56:16.058091 1914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "fc23972cd54e418cb678d9d19faf463e"
format_stamp: "Formatted at 2026-06-21 07:56:16 on dist-test-slave-lbpt"
server_key: "eb89cbaabeccfd9bd563cb4e660cb7d3"
server_key_iv: "4461688260c4929f4cae327a566c0760"
server_key_version: "encryptionkey@0"
I20260621 07:56:16.058259 1914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:16.081580 1914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:16.084730 1914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:16.084905 1914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:16.092036 1914 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:35729
I20260621 07:56:16.092048 1981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:35729 every 8 connection(s)
I20260621 07:56:16.093164 1914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:56:16.096028 1982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:16.101063 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 1914
I20260621 07:56:16.101214 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:56:16.101496 12336 external_mini_cluster.cc:1428] Setting key c1a3e18094e6d7b1ff49e1644c269df9
I20260621 07:56:16.102183 1982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e: Bootstrap starting.
I20260621 07:56:16.104694 1982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:16.105559 1982 log.cc:824] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:16.107817 1982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e: No bootstrap required, opened a new log
I20260621 07:56:16.110942 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:16.102993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42643 (local address 127.12.12.62:35729)
0621 07:56:16.103452 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:16.103471 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:16.103508 (+ 37us) server_negotiation.cc:407] Connection header received
0621 07:56:16.104179 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:16.104198 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:16.104571 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:16.104930 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:16.105267 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:16.106298 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:16.107516 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:16.108039 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:16.108165 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:16.109035 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:16.109040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:16.109053 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:16.109496 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:16.109551 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:16.109575 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:16.109652 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:16.109823 (+ 171us) server_negotiation.cc:299] Negotiation successful
0621 07:56:16.110061 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":141,"threads_started":1}
I20260621 07:56:16.111989 1982 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc23972cd54e418cb678d9d19faf463e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 35729 } }
I20260621 07:56:16.112287 1982 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:56:16.112347 1982 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc23972cd54e418cb678d9d19faf463e, State: Initialized, Role: FOLLOWER
I20260621 07:56:16.112800 1982 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc23972cd54e418cb678d9d19faf463e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 35729 } }
I20260621 07:56:16.112984 1982 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:56:16.113096 1982 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:56:16.113204 1982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:16.114007 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:56:16.114480 1982 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc23972cd54e418cb678d9d19faf463e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 35729 } }
I20260621 07:56:16.114883 1982 leader_election.cc:302] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc23972cd54e418cb678d9d19faf463e; no voters:
I20260621 07:56:16.115295 1982 leader_election.cc:288] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:56:16.115397 1987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:56:16.115677 1987 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [term 1 LEADER]: Becoming Leader. State: Replica: fc23972cd54e418cb678d9d19faf463e, State: Running, Role: LEADER
I20260621 07:56:16.116041 1987 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc23972cd54e418cb678d9d19faf463e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 35729 } }
I20260621 07:56:16.116614 1982 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:16.117288 1987 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc23972cd54e418cb678d9d19faf463e. Latest consensus state: current_term: 1 leader_uuid: "fc23972cd54e418cb678d9d19faf463e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc23972cd54e418cb678d9d19faf463e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 35729 } } }
I20260621 07:56:16.117396 1987 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:16.117790 1991 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc23972cd54e418cb678d9d19faf463e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc23972cd54e418cb678d9d19faf463e" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 35729 } } }
I20260621 07:56:16.117895 1991 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e [sys.catalog]: This master's current role is: LEADER
W20260621 07:56:16.122323 2007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:56:16.122468 2007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:56:16.122560 1999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:56:16.123274 1999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:56:16.125562 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:16.118037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39863 (local address 127.12.12.62:35729)
0621 07:56:16.118168 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:16.118173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:16.118633 (+ 460us) server_negotiation.cc:407] Connection header received
0621 07:56:16.118832 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:16.118835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:16.118907 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:16.118965 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:16.119398 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:16.120529 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:16.121387 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:16.121643 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:16.121753 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:16.125053 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:16.125058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:16.125061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:16.125078 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:16.125111 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:16.125127 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:16.125217 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:16.125356 (+ 139us) server_negotiation.cc:299] Negotiation successful
0621 07:56:16.125421 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 07:56:16.128772 1999 catalog_manager.cc:1347] Generated new cluster ID: bab8950773af4eae950f2baa027b582c
I20260621 07:56:16.128890 1999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:56:16.138554 1999 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:56:16.139807 1999 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:56:16.149307 1999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc23972cd54e418cb678d9d19faf463e: Generated new TSK 0
I20260621 07:56:16.150213 1999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:56:16.158926 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 1914
2026-06-21T07:56:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:56:16 dist-test-slave-lbpt krb5kdc[2013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:56:16 dist-test-slave-lbpt krb5kdc[2013](info): set up 2 sockets
Jun 21 07:56:16 dist-test-slave-lbpt krb5kdc[2013](info): commencing operation
krb5kdc: starting...
W20260621 07:56:17.198932 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:56:17 dist-test-slave-lbpt krb5kdc[2013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:56:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:56:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:56:17.325721 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:34493
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:34493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:17.436642 2026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:17.436928 2026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:17.436969 2026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:17.440989 2026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:17.441056 2026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:17.441076 2026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:17.441093 2026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:17.441109 2026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:17.446535 2026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:34493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:34493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:17.447618 2026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:17.448657 2026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:17.457331 2031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:17.457376 2026 server_base.cc:1034] running on GCE node
W20260621 07:56:17.457383 2034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:17.457473 2032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:17.458196 2026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:17.459446 2026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:17.460644 2026 hybrid_clock.cc:648] HybridClock initialized: now 1782028577460629 us; error 28 us; skew 500 ppm
Jun 21 07:56:17 dist-test-slave-lbpt krb5kdc[2013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028577, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:17.463646 2026 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:56:17.464603 2026 webserver.cc:460] Webserver started at http://127.12.12.62:40907/ using document root <none> and password file <none>
I20260621 07:56:17.465173 2026 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:17.465301 2026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:17.465541 2026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:17.467701 2026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "009ff15231d94435b3555a9ab2711dfd"
format_stamp: "Formatted at 2026-06-21 07:56:17 on dist-test-slave-lbpt"
server_key: "e6fbd5eb278ceccafcaf34890b368fb6"
server_key_iv: "eb7ccf1eb2cfaba3f211efb253457199"
server_key_version: "encryptionkey@0"
I20260621 07:56:17.468272 2026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "009ff15231d94435b3555a9ab2711dfd"
format_stamp: "Formatted at 2026-06-21 07:56:17 on dist-test-slave-lbpt"
server_key: "e6fbd5eb278ceccafcaf34890b368fb6"
server_key_iv: "eb7ccf1eb2cfaba3f211efb253457199"
server_key_version: "encryptionkey@0"
I20260621 07:56:17.472018 2026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:56:17.474644 2041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:17.475956 2026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:56:17.476114 2026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "009ff15231d94435b3555a9ab2711dfd"
format_stamp: "Formatted at 2026-06-21 07:56:17 on dist-test-slave-lbpt"
server_key: "e6fbd5eb278ceccafcaf34890b368fb6"
server_key_iv: "eb7ccf1eb2cfaba3f211efb253457199"
server_key_version: "encryptionkey@0"
I20260621 07:56:17.476235 2026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:17.480973 2026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:17.484082 2026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:17.484287 2026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:17.491143 2026 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:34493
I20260621 07:56:17.491171 2093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:34493 every 8 connection(s)
I20260621 07:56:17.492131 2026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:56:17.494918 2094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:17.500458 2094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd: Bootstrap starting.
I20260621 07:56:17.502189 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 2026
I20260621 07:56:17.502332 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:56:17.502665 12336 external_mini_cluster.cc:1428] Setting key ccd1ffc10da6c6e0d6851ea3211ca59c
I20260621 07:56:17.503266 2094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:17.504233 2094 log.cc:824] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:17.506242 2094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd: No bootstrap required, opened a new log
I20260621 07:56:17.508034 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:17.503852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36787 (local address 127.12.12.62:34493)
0621 07:56:17.504346 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:17.504393 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:17.504440 (+ 47us) server_negotiation.cc:407] Connection header received
0621 07:56:17.505347 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:17.505368 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:17.505653 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:17.506019 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:17.506836 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:17.506840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:17.506853 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:17.506861 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:17.506907 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:17.506925 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:17.507044 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:17.507150 (+ 106us) server_negotiation.cc:299] Negotiation successful
0621 07:56:17.507314 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":136,"threads_started":1}
I20260621 07:56:17.510281 2094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009ff15231d94435b3555a9ab2711dfd" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34493 } }
I20260621 07:56:17.510615 2094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:56:17.510740 2094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009ff15231d94435b3555a9ab2711dfd, State: Initialized, Role: FOLLOWER
I20260621 07:56:17.511162 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:56:17.511281 2094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009ff15231d94435b3555a9ab2711dfd" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34493 } }
I20260621 07:56:17.511448 2094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:56:17.511513 2094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:56:17.511610 2094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:17.512694 2094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009ff15231d94435b3555a9ab2711dfd" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34493 } }
I20260621 07:56:17.513057 2094 leader_election.cc:302] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 009ff15231d94435b3555a9ab2711dfd; no voters:
I20260621 07:56:17.513404 2094 leader_election.cc:288] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:56:17.513482 2103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:56:17.513666 2103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [term 1 LEADER]: Becoming Leader. State: Replica: 009ff15231d94435b3555a9ab2711dfd, State: Running, Role: LEADER
I20260621 07:56:17.514008 2103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009ff15231d94435b3555a9ab2711dfd" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34493 } }
I20260621 07:56:17.514837 2094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:17.515431 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:17.513326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43717 (local address 127.12.12.62:34493)
0621 07:56:17.513464 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:17.513469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:17.513854 (+ 385us) server_negotiation.cc:407] Connection header received
0621 07:56:17.513894 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:17.513899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:17.513937 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:17.514002 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:17.514875 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:17.514879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:17.514882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:56:17.514893 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:17.514922 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:56:17.514937 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:17.515036 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:17.515273 (+ 237us) server_negotiation.cc:299] Negotiation successful
0621 07:56:17.515315 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:56:17.515694 2103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "009ff15231d94435b3555a9ab2711dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009ff15231d94435b3555a9ab2711dfd" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34493 } } }
I20260621 07:56:17.515828 2103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:17.516392 2105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 009ff15231d94435b3555a9ab2711dfd. Latest consensus state: current_term: 1 leader_uuid: "009ff15231d94435b3555a9ab2711dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009ff15231d94435b3555a9ab2711dfd" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 34493 } } }
I20260621 07:56:17.516731 2105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:17.516810 2115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:56:17.519979 2115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:56:17.525385 2115 catalog_manager.cc:1347] Generated new cluster ID: 1bab25fe5e674e0f97403c54fd0b8cb0
I20260621 07:56:17.525480 2115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:56:17.547423 2115 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:56:17.547822 2115 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:56:17.556067 2115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 009ff15231d94435b3555a9ab2711dfd: Generated new TSK 0
I20260621 07:56:17.557255 2115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:56:17.562738 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 2026
2026-06-21T07:56:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1403 ms)
[----------] 18 tests from AuthTokenIssuingTest (25743 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:56:17 dist-test-slave-lbpt krb5kdc[2125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:56:17 dist-test-slave-lbpt krb5kdc[2125](info): set up 2 sockets
Jun 21 07:56:17 dist-test-slave-lbpt krb5kdc[2125](info): commencing operation
krb5kdc: starting...
W20260621 07:56:18.622833 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:56:18 dist-test-slave-lbpt krb5kdc[2125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:56:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:56:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:56:18.745445 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:38159
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:38159,127.12.12.61:40065,127.12.12.60:40523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:18.855719 2138 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:56:18.856012 2138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:18.856036 2138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:18.856133 2138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:18.860177 2138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:18.860232 2138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:18.860251 2138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:18.860268 2138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:18.860284 2138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:18.866243 2138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:38159,127.12.12.61:40065,127.12.12.60:40523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:38159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:18.867444 2138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:18.868482 2138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:18.877238 2144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:18.877265 2143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:18.877308 2138 server_base.cc:1034] running on GCE node
W20260621 07:56:18.877238 2146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:18.878116 2138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:18.879231 2138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:18.880419 2138 hybrid_clock.cc:648] HybridClock initialized: now 1782028578880407 us; error 28 us; skew 500 ppm
Jun 21 07:56:18 dist-test-slave-lbpt krb5kdc[2125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028578, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:18.886368 2138 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:56:18.887387 2138 webserver.cc:460] Webserver started at http://127.12.12.62:40277/ using document root <none> and password file <none>
I20260621 07:56:18.888008 2138 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:18.888106 2138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:18.888342 2138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:18.890475 2138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "5dc109126329457e8f01aa9357f41fa3"
format_stamp: "Formatted at 2026-06-21 07:56:18 on dist-test-slave-lbpt"
server_key: "ba8ebddd770f0cbc8ad640c38061a575"
server_key_iv: "e026fcf11fc77f936977474aea2cd26e"
server_key_version: "encryptionkey@0"
I20260621 07:56:18.891055 2138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "5dc109126329457e8f01aa9357f41fa3"
format_stamp: "Formatted at 2026-06-21 07:56:18 on dist-test-slave-lbpt"
server_key: "ba8ebddd770f0cbc8ad640c38061a575"
server_key_iv: "e026fcf11fc77f936977474aea2cd26e"
server_key_version: "encryptionkey@0"
I20260621 07:56:18.894945 2138 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 07:56:18.897576 2153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:18.899017 2138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:56:18.899169 2138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "5dc109126329457e8f01aa9357f41fa3"
format_stamp: "Formatted at 2026-06-21 07:56:18 on dist-test-slave-lbpt"
server_key: "ba8ebddd770f0cbc8ad640c38061a575"
server_key_iv: "e026fcf11fc77f936977474aea2cd26e"
server_key_version: "encryptionkey@0"
I20260621 07:56:18.899289 2138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:18.927179 2138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:18.930456 2138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:18.930719 2138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:18.937953 2138 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:38159
I20260621 07:56:18.938249 2205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:38159 every 8 connection(s)
I20260621 07:56:18.938990 2138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:56:18.941793 2206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:18.942255 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 2138
I20260621 07:56:18.942401 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:56:18.942780 12336 external_mini_cluster.cc:1428] Setting key 90a497f75d252696a0fc6ae9aa4b8f5f
I20260621 07:56:18.945664 2206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.12.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.61@KRBTEST.COM" created.
Jun 21 07:56:18 dist-test-slave-lbpt krb5kdc[2125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028578, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:56:18.961267 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:18.948142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.12.12.62:38159)
0621 07:56:18.948563 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:18.948582 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:18.948856 (+ 274us) server_negotiation.cc:407] Connection header received
0621 07:56:18.949902 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:18.949906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:18.950077 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:18.950270 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:18.951137 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:18.951985 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:18.952943 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:18.953302 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:18.953466 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:18.956146 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:18.956156 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:18.956172 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:18.956204 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:18.958376 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:18.958896 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:18.958902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:18.958904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:18.959058 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:18.959359 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:18.959363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:18.959364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:18.959897 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:18.960046 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:18.960386 (+ 340us) server_negotiation.cc:299] Negotiation successful
0621 07:56:18.960619 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":282,"thread_start_us":145,"threads_started":1}
I20260621 07:56:18.961266 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:18.948143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38159 (local address 127.0.0.1:50760)
0621 07:56:18.948709 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:18.948751 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:18.949591 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:18.950256 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:18.950261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:18.950756 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:18.950995 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:18.951008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:18.952107 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:18.952110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:18.952828 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:18.952836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:18.953450 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:18.953454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:18.953561 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:18.954180 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:18.954195 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:18.955978 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:18.958518 (+ 2540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:18.958526 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:18.958533 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:18.958749 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:18.959156 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:18.959158 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:18.959159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:18.959263 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:18.960018 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:18.960023 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:18.960139 (+ 116us) client_negotiation.cc:769] Sending connection context
0621 07:56:18.960270 (+ 131us) client_negotiation.cc:240] Negotiation successful
0621 07:56:18.960514 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"spinlock_wait_cycles":31488,"thread_start_us":151,"threads_started":1}
I20260621 07:56:18.963891 2206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:56:18.964618 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:18.964351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40065 (local address 127.0.0.1:53784)
0621 07:56:18.964479 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:18.964557 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260621 07:56:18.964833 2154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40065: connect: Connection refused (error 111)
I20260621 07:56:18.965848 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:18.965631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40065 (local address 127.0.0.1:53800)
0621 07:56:18.965740 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:18.965793 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 07:56:18.966045 2206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.61:40065: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40065: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.61.keytab.
Entry for principal kudu/127.12.12.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.61.keytab.
I20260621 07:56:18.969678 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.61:40065
--webserver_interface=127.12.12.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.61.keytab
--principal=kudu/127.12.12.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:38159,127.12.12.61:40065,127.12.12.60:40523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:56:19.015290 2206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } attempt: 1
I20260621 07:56:19.016355 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.015989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40065 (local address 127.0.0.1:53806)
0621 07:56:19.016197 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.016260 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260621 07:56:19.017132 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.016902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40065 (local address 127.0.0.1:53822)
0621 07:56:19.017017 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.017071 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 07:56:19.017262 2206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.61:40065: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40065: connect: Connection refused (error 111)
W20260621 07:56:19.083357 2213 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:56:19.083691 2213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:19.083770 2213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:19.083863 2213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260621 07:56:19.085448 2206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } attempt: 2
I20260621 07:56:19.086277 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.085989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40065 (local address 127.0.0.1:53828)
0621 07:56:19.086137 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.086205 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260621 07:56:19.086936 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.086741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40065 (local address 127.0.0.1:53836)
0621 07:56:19.086838 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.086890 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260621 07:56:19.087047 2206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.61:40065: Network error: Client connection negotiation failed: client connection to 127.12.12.61:40065: connect: Connection refused (error 111)
W20260621 07:56:19.088284 2213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:19.088361 2213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:19.088425 2213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:19.088465 2213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:19.088497 2213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:19.093879 2213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:38159,127.12.12.61:40065,127.12.12.60:40523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.61.keytab
--principal=kudu/127.12.12.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.61:40065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.12.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:19.095093 2213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:19.096138 2213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:19.104933 2219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:19.104969 2218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:19.105134 2213 server_base.cc:1034] running on GCE node
W20260621 07:56:19.104933 2221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:19.105854 2213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:19.107110 2213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:19.108343 2213 hybrid_clock.cc:648] HybridClock initialized: now 1782028579108326 us; error 32 us; skew 500 ppm
Jun 21 07:56:19 dist-test-slave-lbpt krb5kdc[2125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028579, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:19.111339 2213 init.cc:377] Logged in from keytab as kudu/127.12.12.61@KRBTEST.COM (short username kudu)
I20260621 07:56:19.112346 2213 webserver.cc:460] Webserver started at http://127.12.12.61:35275/ using document root <none> and password file <none>
I20260621 07:56:19.112926 2213 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:19.113018 2213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:19.113237 2213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:19.115459 2213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/data/instance:
uuid: "4b9a1a45b38a4293a33afff4b45c737d"
format_stamp: "Formatted at 2026-06-21 07:56:19 on dist-test-slave-lbpt"
server_key: "614f209799842037ee8ff5c7b832ede4"
server_key_iv: "31b76153755c7cbce908cf09099abfed"
server_key_version: "encryptionkey@0"
I20260621 07:56:19.116045 2213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/wal/instance:
uuid: "4b9a1a45b38a4293a33afff4b45c737d"
format_stamp: "Formatted at 2026-06-21 07:56:19 on dist-test-slave-lbpt"
server_key: "614f209799842037ee8ff5c7b832ede4"
server_key_iv: "31b76153755c7cbce908cf09099abfed"
server_key_version: "encryptionkey@0"
I20260621 07:56:19.119958 2213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:56:19.122648 2228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:19.124008 2213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:56:19.124167 2213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/wal
uuid: "4b9a1a45b38a4293a33afff4b45c737d"
format_stamp: "Formatted at 2026-06-21 07:56:19 on dist-test-slave-lbpt"
server_key: "614f209799842037ee8ff5c7b832ede4"
server_key_iv: "31b76153755c7cbce908cf09099abfed"
server_key_version: "encryptionkey@0"
I20260621 07:56:19.124292 2213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:19.135499 2213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:19.138732 2213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:19.138950 2213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:19.146185 2213 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.61:40065
I20260621 07:56:19.146519 2280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.61:40065 every 8 connection(s)
I20260621 07:56:19.147291 2213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/data/info.pb
I20260621 07:56:19.150084 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:19.153925 2281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:56:19.156281 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 2213
I20260621 07:56:19.156414 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-1/wal/instance
I20260621 07:56:19.156697 12336 external_mini_cluster.cc:1428] Setting key 4b650abdb3ae0a1dc4a5dfed9218c7ce
Jun 21 07:56:19 dist-test-slave-lbpt krb5kdc[2125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028579, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.61@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.60@KRBTEST.COM" created.
I20260621 07:56:19.170452 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.155924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50774 (local address 127.12.12.62:38159)
0621 07:56:19.156102 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.156107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.157034 (+ 927us) server_negotiation.cc:407] Connection header received
0621 07:56:19.158018 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.158024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.158096 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.158214 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.159894 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.160443 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.162334 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.162674 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.162778 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.165773 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:19.165776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:19.165778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:19.165807 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:19.168141 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.168787 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.168791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.168793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.168889 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.169265 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.169268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.169270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.169472 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:19.169580 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:19.170126 (+ 546us) server_negotiation.cc:299] Negotiation successful
0621 07:56:19.170259 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:56:19.171324 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.156272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38159 (local address 127.0.0.1:50774)
0621 07:56:19.156848 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.156902 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:19.157763 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:19.158474 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:19.158487 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:19.159005 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:19.159724 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:19.159747 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.160585 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.160592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:19.162175 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:19.162191 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.162877 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.162884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:19.163101 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.163792 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:19.163818 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:19.165585 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:19.168296 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:19.168305 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:19.168326 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:19.168665 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:19.168991 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:19.168994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:19.168996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:19.169153 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:19.169579 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:19.169588 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:19.169864 (+ 276us) client_negotiation.cc:769] Sending connection context
0621 07:56:19.170072 (+ 208us) client_negotiation.cc:240] Negotiation successful
0621 07:56:19.170461 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":170,"threads_started":1}
I20260621 07:56:19.172894 2281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:56:19.178202 2206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } attempt: 3
Jun 21 07:56:19 dist-test-slave-lbpt krb5kdc[2125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028579, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.61@KRBTEST.COM for kudu/127.12.12.61@KRBTEST.COM
I20260621 07:56:19.183965 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.173452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40065 (local address 127.0.0.1:53852)
0621 07:56:19.173571 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.173635 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:19.173700 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:19.174324 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:19.174329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:19.174540 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:19.174663 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:19.174673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.175666 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.175671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:19.176633 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:19.176646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.177226 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.177230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:19.177386 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.177888 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:19.177900 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:19.179800 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:19.182164 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:19.182169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:19.182171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:19.182371 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:19.182734 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:19.182741 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:19.182742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:19.182817 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:19.183415 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:19.183417 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:19.183492 (+ 75us) client_negotiation.cc:769] Sending connection context
0621 07:56:19.183608 (+ 116us) client_negotiation.cc:240] Negotiation successful
0621 07:56:19.183753 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":63}
Jun 21 07:56:19 dist-test-slave-lbpt krb5kdc[2125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028578, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for kudu/127.12.12.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.60.keytab.
Entry for principal kudu/127.12.12.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.60.keytab.
I20260621 07:56:19.184844 2286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.173612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.12.12.61:40065)
0621 07:56:19.173974 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.173982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.173998 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:56:19.174030 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.174034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.174182 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.174284 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.174797 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.175543 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.176755 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.177116 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.177302 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.179933 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:19.179947 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:19.179968 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:19.179994 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:19.182051 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.182541 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.182547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.182549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.182636 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.182928 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.182934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.182937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.183281 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:19.183423 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:19.184238 (+ 815us) server_negotiation.cc:299] Negotiation successful
0621 07:56:19.184421 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":281,"thread_start_us":179,"threads_started":1}
I20260621 07:56:19.185715 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.60:40523
--webserver_interface=127.12.12.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.60.keytab
--principal=kudu/127.12.12.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:38159,127.12.12.61:40065,127.12.12.60:40523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:56:19.187876 2281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:56:19.188633 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.188391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:40523 (local address 127.0.0.1:39054)
0621 07:56:19.188492 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.188573 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260621 07:56:19.188637 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.178754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40065 (local address 127.0.0.1:53858)
0621 07:56:19.178923 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.178939 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:19.179034 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:19.179493 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:19.179496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:19.179737 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:19.179870 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:19.179878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.180886 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.180891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:19.181822 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:19.181837 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.182573 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.182577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:19.182759 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.183173 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:19.183187 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:19.184714 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:19.186883 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:19.186886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:19.186888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:19.187179 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:19.187508 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:19.187512 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:19.187514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:19.187608 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:19.188058 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:19.188062 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:19.188164 (+ 102us) client_negotiation.cc:769] Sending connection context
0621 07:56:19.188252 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 07:56:19.188380 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
W20260621 07:56:19.188853 2232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
I20260621 07:56:19.189406 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.178932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53858 (local address 127.12.12.61:40065)
0621 07:56:19.179277 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.179284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.179295 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:56:19.179325 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.179327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.179360 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.179465 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.179999 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.180773 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.182039 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.182364 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.182560 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.185129 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:19.185134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:19.185137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:19.185157 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:19.186759 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.187275 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.187279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.187281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.187377 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.187713 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.187716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.187718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.187941 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:19.188930 (+ 989us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:19.189011 (+ 81us) server_negotiation.cc:299] Negotiation successful
0621 07:56:19.189139 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":97,"threads_started":1}
I20260621 07:56:19.190004 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.189517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:40523 (local address 127.0.0.1:39056)
0621 07:56:19.189814 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.189878 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":116,"threads_started":1}
W20260621 07:56:19.190160 2281 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:40523: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
I20260621 07:56:19.190192 2206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:56:19.190969 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.190655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:40523 (local address 127.0.0.1:39060)
0621 07:56:19.190812 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.190876 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260621 07:56:19.191694 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.191389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:40523 (local address 127.0.0.1:39062)
0621 07:56:19.191559 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.191620 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260621 07:56:19.191888 2206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:40523: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
I20260621 07:56:19.223076 2206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } attempt: 1
I20260621 07:56:19.223989 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.223646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:40523 (local address 127.0.0.1:39076)
0621 07:56:19.223819 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.223895 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260621 07:56:19.224756 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.224540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:40523 (local address 127.0.0.1:39080)
0621 07:56:19.224643 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.224698 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260621 07:56:19.224910 2206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:40523: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
I20260621 07:56:19.239461 2281 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } attempt: 1
I20260621 07:56:19.240576 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.240256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:40523 (local address 127.0.0.1:39096)
0621 07:56:19.240435 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.240499 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260621 07:56:19.241515 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.241283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:40523 (local address 127.0.0.1:39108)
0621 07:56:19.241413 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.241461 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260621 07:56:19.241670 2281 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:40523: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
I20260621 07:56:19.300104 2206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } attempt: 2
I20260621 07:56:19.301040 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.300713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:40523 (local address 127.0.0.1:39120)
0621 07:56:19.300908 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.300965 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260621 07:56:19.301582 2288 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260621 07:56:19.301815 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.301596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:40523 (local address 127.0.0.1:39128)
0621 07:56:19.301705 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.301758 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 07:56:19.301884 2288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:19.301944 2288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:19.302004 2288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:19.302070 2206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:40523: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
W20260621 07:56:19.306118 2288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:19.306174 2288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:19.306195 2288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:19.306210 2288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:19.306226 2288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:19.309865 2281 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } attempt: 2
I20260621 07:56:19.310669 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.310363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:40523 (local address 127.0.0.1:39138)
0621 07:56:19.310544 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.310602 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260621 07:56:19.311447 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.311175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:40523 (local address 127.0.0.1:39154)
0621 07:56:19.311352 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.311398 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260621 07:56:19.311726 2281 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:40523: Network error: Client connection negotiation failed: client connection to 127.12.12.60:40523: connect: Connection refused (error 111)
I20260621 07:56:19.312074 2288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:38159,127.12.12.61:40065,127.12.12.60:40523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.60.keytab
--principal=kudu/127.12.12.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.60:40523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.12.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:19.313138 2288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:19.314152 2288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:19.322615 2296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:19.322786 2288 server_base.cc:1034] running on GCE node
W20260621 07:56:19.322857 2298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:19.322863 2295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:19.323446 2288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:19.324534 2288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:19.325690 2288 hybrid_clock.cc:648] HybridClock initialized: now 1782028579325680 us; error 25 us; skew 500 ppm
Jun 21 07:56:19 dist-test-slave-lbpt krb5kdc[2125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028579, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:19.328763 2288 init.cc:377] Logged in from keytab as kudu/127.12.12.60@KRBTEST.COM (short username kudu)
I20260621 07:56:19.329682 2288 webserver.cc:460] Webserver started at http://127.12.12.60:36687/ using document root <none> and password file <none>
I20260621 07:56:19.330302 2288 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:19.330400 2288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:19.330636 2288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:19.332757 2288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/data/instance:
uuid: "f16df8c809b64b89ac673a4b63f3507d"
format_stamp: "Formatted at 2026-06-21 07:56:19 on dist-test-slave-lbpt"
server_key: "74670984870f3646f4aa2b35e6139b39"
server_key_iv: "78c6e6e6cb9bfa5f29355777c0a83b8b"
server_key_version: "encryptionkey@0"
I20260621 07:56:19.333341 2288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/wal/instance:
uuid: "f16df8c809b64b89ac673a4b63f3507d"
format_stamp: "Formatted at 2026-06-21 07:56:19 on dist-test-slave-lbpt"
server_key: "74670984870f3646f4aa2b35e6139b39"
server_key_iv: "78c6e6e6cb9bfa5f29355777c0a83b8b"
server_key_version: "encryptionkey@0"
I20260621 07:56:19.337090 2288 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260621 07:56:19.339476 2305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:19.340629 2288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:56:19.340773 2288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/wal
uuid: "f16df8c809b64b89ac673a4b63f3507d"
format_stamp: "Formatted at 2026-06-21 07:56:19 on dist-test-slave-lbpt"
server_key: "74670984870f3646f4aa2b35e6139b39"
server_key_iv: "78c6e6e6cb9bfa5f29355777c0a83b8b"
server_key_version: "encryptionkey@0"
I20260621 07:56:19.340922 2288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:19.370661 2288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:19.374567 2288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:19.374789 2288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:19.382329 2288 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.60:40523
I20260621 07:56:19.382627 2357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.60:40523 every 8 connection(s)
I20260621 07:56:19.383528 2288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/data/info.pb
I20260621 07:56:19.386644 2358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:19.390623 2358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:56:19.393523 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 2288
I20260621 07:56:19.393675 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028299065733-12336-0/minicluster-data/master-2/wal/instance
I20260621 07:56:19.394003 12336 external_mini_cluster.cc:1428] Setting key 5e4d23aead251c6cde80011fcc39b113
I20260621 07:56:19.401237 2206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } attempt: 3
Jun 21 07:56:19 dist-test-slave-lbpt krb5kdc[2125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028579, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.60@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
Jun 21 07:56:19 dist-test-slave-lbpt krb5kdc[2125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.60@KRBTEST.COM
I20260621 07:56:19.402897 2281 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } attempt: 3
I20260621 07:56:19.407148 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.392553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.12.12.62:38159)
0621 07:56:19.392703 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.392707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.393714 (+ 1007us) server_negotiation.cc:407] Connection header received
0621 07:56:19.395091 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.395095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.395163 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.395220 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.396588 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.397436 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.398805 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.399129 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.399246 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.402543 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:19.402547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:19.402550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:19.402576 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:19.404512 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.405201 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.405205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.405207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.405304 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.405719 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.405724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.405727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.406015 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:19.406111 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:19.406843 (+ 732us) server_negotiation.cc:299] Negotiation successful
0621 07:56:19.406958 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Jun 21 07:56:19 dist-test-slave-lbpt krb5kdc[2125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028578, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for kudu/127.12.12.60@KRBTEST.COM
I20260621 07:56:19.408773 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.392938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:38159 (local address 127.0.0.1:50776)
0621 07:56:19.393534 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.393596 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:19.394814 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:19.395513 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:19.395520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:19.396002 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:19.396382 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:19.396420 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.397557 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.397559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:19.398684 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:19.398696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.399305 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.399308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:19.399459 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.400089 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:19.400115 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:19.402337 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:19.404632 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:19.404642 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:19.404653 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:19.405065 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:19.405421 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:19.405424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:19.405426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:19.405592 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:19.406176 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:19.406185 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:19.406537 (+ 352us) client_negotiation.cc:769] Sending connection context
0621 07:56:19.407366 (+ 829us) client_negotiation.cc:240] Negotiation successful
0621 07:56:19.407697 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":136,"threads_started":1}
I20260621 07:56:19.408942 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.394769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.12.12.60:40523)
0621 07:56:19.395134 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.395143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.395179 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:56:19.395403 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.395407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.395573 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.395716 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.396593 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.397775 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.398947 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.399475 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.399555 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.402917 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:19.402926 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:19.402940 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:19.402964 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:19.406548 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.407073 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.407080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.407082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.407167 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.407492 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.407495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.407496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.407750 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:19.407950 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:19.408597 (+ 647us) server_negotiation.cc:299] Negotiation successful
0621 07:56:19.408737 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":158,"threads_started":1}
Jun 21 07:56:19 dist-test-slave-lbpt krb5kdc[2125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028579, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.61@KRBTEST.COM for kudu/127.12.12.60@KRBTEST.COM
I20260621 07:56:19.410528 2358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:56:19.412911 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.401729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:40523 (local address 127.0.0.1:39180)
0621 07:56:19.401905 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.401925 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:19.402011 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:19.402455 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:19.402459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:19.402657 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:19.402764 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:19.402770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.403945 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.403949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:19.404811 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:19.404821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.405547 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.405551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:19.405638 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.406079 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:19.406097 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:19.407852 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:19.411290 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:19.411293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:19.411294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:19.411555 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:19.411879 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:19.411883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:19.411886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:19.411984 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:19.412437 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:19.412440 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:19.412534 (+ 94us) client_negotiation.cc:769] Sending connection context
0621 07:56:19.412631 (+ 97us) client_negotiation.cc:240] Negotiation successful
0621 07:56:19.412756 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260621 07:56:19.413493 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.401786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39180 (local address 127.12.12.60:40523)
0621 07:56:19.402201 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.402212 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.402225 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:56:19.402262 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.402264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.402307 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.402403 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.402892 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.403775 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.404925 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.405335 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.405436 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.407985 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:19.407989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:19.407992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:19.408015 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:19.411162 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.411656 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.411660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.411663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.411763 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.412091 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.412095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.412096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.412296 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:19.412395 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:19.413183 (+ 788us) server_negotiation.cc:299] Negotiation successful
0621 07:56:19.413296 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":340,"thread_start_us":127,"threads_started":1}
I20260621 07:56:19.416755 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.404046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:40523 (local address 127.0.0.1:39184)
0621 07:56:19.404261 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.404276 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:19.404372 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:19.404605 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:19.404610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:19.404808 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:19.404974 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:19.404981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.406278 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.406282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:19.407222 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:19.407237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.407858 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.407862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:19.407986 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.408659 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:19.408675 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:19.410188 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:19.415012 (+ 4824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:19.415015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:19.415017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:19.415285 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:19.415648 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:19.415654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:19.415658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:19.415744 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:19.416188 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:19.416192 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:19.416302 (+ 110us) client_negotiation.cc:769] Sending connection context
0621 07:56:19.416420 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 07:56:19.416556 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
Jun 21 07:56:19 dist-test-slave-lbpt krb5kdc[2125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028579, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.60@KRBTEST.COM for kudu/127.12.12.61@KRBTEST.COM
I20260621 07:56:19.416976 2206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3: Bootstrap starting.
I20260621 07:56:19.417434 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.403611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.12.12.60:40523)
0621 07:56:19.404101 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.404112 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.404328 (+ 216us) server_negotiation.cc:407] Connection header received
0621 07:56:19.404446 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.404450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.404486 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.404586 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.405103 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.406142 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.407386 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.407705 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.409287 (+ 1582us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.410354 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:19.410359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:19.410365 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:19.410392 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:19.414905 (+ 4513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.415427 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.415431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.415434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.415530 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.415865 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.415870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.415872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.416072 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:19.417042 (+ 970us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:19.417118 (+ 76us) server_negotiation.cc:299] Negotiation successful
0621 07:56:19.417221 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":421,"spinlock_wait_cycles":3328,"thread_start_us":94,"threads_started":1}
Jun 21 07:56:19 dist-test-slave-lbpt krb5kdc[2125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:56:19.420192 2281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d: Bootstrap starting.
I20260621 07:56:19.421494 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.411721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.12.12.61:40065)
0621 07:56:19.411873 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.411877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.411889 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:56:19.411924 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.411930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.411986 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.412065 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.412510 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.413223 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.414300 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.414698 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.414784 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.417327 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:19.417331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:19.417333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:19.417356 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:19.419352 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.420148 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.420152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.420154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.420241 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.420565 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.420568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.420572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.420841 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:19.420958 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:19.421250 (+ 292us) server_negotiation.cc:299] Negotiation successful
0621 07:56:19.421377 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:56:19.421981 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.411244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:40065 (local address 127.0.0.1:53860)
0621 07:56:19.411718 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.411735 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:19.411802 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:19.412103 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:19.412106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:19.412280 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:19.412408 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:19.412414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.413340 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.413343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:19.414163 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:19.414175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.414953 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.414957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:19.415070 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.415528 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:19.415541 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:19.417186 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:19.419481 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:19.419487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:19.419490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:19.419739 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:19.420371 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:19.420373 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:19.420375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:19.420461 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:19.420948 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:19.420951 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:19.421053 (+ 102us) client_negotiation.cc:769] Sending connection context
0621 07:56:19.421677 (+ 624us) client_negotiation.cc:240] Negotiation successful
0621 07:56:19.421805 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"spinlock_wait_cycles":1792}
I20260621 07:56:19.422478 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.411797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.12.12.62:38159)
0621 07:56:19.411906 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.411911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.412078 (+ 167us) server_negotiation.cc:407] Connection header received
0621 07:56:19.412234 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.412237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.412278 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.412361 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.413221 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.413833 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.414891 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.415165 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.415233 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.418537 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:19.418543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:19.418549 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:19.418577 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:19.420702 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.421162 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.421167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.421169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.421252 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.421568 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.421572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.421573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.421787 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:19.422000 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:19.422182 (+ 182us) server_negotiation.cc:299] Negotiation successful
0621 07:56:19.422296 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 07:56:19.422695 2358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:56:19.422992 2281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:19.424955 2206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:19.425848 2206 log.cc:824] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:19.427098 2281 log.cc:824] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:56:19 dist-test-slave-lbpt krb5kdc[2125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028579, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.60@KRBTEST.COM for kudu/127.12.12.60@KRBTEST.COM
Jun 21 07:56:19 dist-test-slave-lbpt krb5kdc[2125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.61@KRBTEST.COM
I20260621 07:56:19.430090 2281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d: No bootstrap required, opened a new log
I20260621 07:56:19.431243 2206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3: No bootstrap required, opened a new log
I20260621 07:56:19.431686 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.423144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:40523 (local address 127.0.0.1:39196)
0621 07:56:19.423267 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:19.423283 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:19.423366 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:19.423610 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:19.423613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:19.423791 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:19.423872 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:19.423879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.424769 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.424775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:19.425355 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:19.425363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.425829 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.425832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:19.425921 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.426380 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:19.426394 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:19.427950 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:19.429969 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:19.429972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:19.429973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:19.430169 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:19.430541 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:19.430545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:19.430546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:19.430641 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:19.431098 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:19.431101 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:19.431217 (+ 116us) client_negotiation.cc:769] Sending connection context
0621 07:56:19.431327 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 07:56:19.431489 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 07:56:19.432147 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.423333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.12.12.60:40523)
0621 07:56:19.423436 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.423441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.423451 (+ 10us) server_negotiation.cc:407] Connection header received
0621 07:56:19.423477 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.423479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.423520 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.423602 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.423982 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.424666 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.425460 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.425722 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.425835 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.428150 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:19.428156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:19.428162 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:19.428189 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:19.429749 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.430252 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.430257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.430259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.430335 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.430758 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.430764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.430768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.430997 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:19.431110 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:19.431882 (+ 772us) server_negotiation.cc:299] Negotiation successful
0621 07:56:19.431996 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 07:56:19.434177 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.424109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.12.12.61:40065)
0621 07:56:19.424201 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.424205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.424216 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:56:19.424248 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.424251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.424286 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.424362 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.425110 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.425825 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.426929 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.427250 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.427354 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.430342 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:19.430346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:19.430349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:19.430374 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:19.432097 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.432762 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.432766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.432767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.432851 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.433149 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.433152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.433154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.433370 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:19.433534 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:19.433907 (+ 373us) server_negotiation.cc:299] Negotiation successful
0621 07:56:19.434029 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 07:56:19.434824 2358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d: Bootstrap starting.
I20260621 07:56:19.435603 2206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } }
I20260621 07:56:19.435835 2206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc109126329457e8f01aa9357f41fa3, State: Initialized, Role: FOLLOWER
I20260621 07:56:19.436158 2206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } }
I20260621 07:56:19.436609 2367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } } }
I20260621 07:56:19.436753 2367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:56:19.437124 2358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:19.437438 2206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:19.437855 2358 log.cc:824] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:19.439749 2358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d: No bootstrap required, opened a new log
I20260621 07:56:19.441845 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:56:19.443334 2180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5dc109126329457e8f01aa9357f41fa3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50784
I20260621 07:56:19.443369 2281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } }
I20260621 07:56:19.443614 2281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9a1a45b38a4293a33afff4b45c737d, State: Initialized, Role: FOLLOWER
I20260621 07:56:19.443599 2180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 07:56:19.443715 2180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:19.443965 2358 raft_consensus.cc:348] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } }
I20260621 07:56:19.443993 2281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } }
I20260621 07:56:19.444178 2358 raft_consensus.cc:729] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f16df8c809b64b89ac673a4b63f3507d, State: Initialized, Role: FOLLOWER
I20260621 07:56:19.444505 2358 consensus_queue.cc:260] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } }
I20260621 07:56:19.444538 2373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } } }
I20260621 07:56:19.444777 2373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:56:19.445083 2180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } }
I20260621 07:56:19.445456 2281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:19.445761 2358 sys_catalog.cc:564] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:19.446028 2180 leader_election.cc:288] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [CANDIDATE]: Term 1 election: Requested vote from peers 4b9a1a45b38a4293a33afff4b45c737d (127.12.12.61:40065), f16df8c809b64b89ac673a4b63f3507d (127.12.12.60:40523)
I20260621 07:56:19.446826 2255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5dc109126329457e8f01aa9357f41fa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4b9a1a45b38a4293a33afff4b45c737d"
I20260621 07:56:19.446909 2332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5dc109126329457e8f01aa9357f41fa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f16df8c809b64b89ac673a4b63f3507d"
I20260621 07:56:19.447044 2255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:19.447081 2332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:19.448423 2255 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dc109126329457e8f01aa9357f41fa3 in term 1.
I20260621 07:56:19.448423 2332 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dc109126329457e8f01aa9357f41fa3 in term 1.
I20260621 07:56:19.448848 2157 leader_election.cc:302] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dc109126329457e8f01aa9357f41fa3, f16df8c809b64b89ac673a4b63f3507d; no voters:
I20260621 07:56:19.449399 2367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:56:19.449632 2367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [term 1 LEADER]: Becoming Leader. State: Replica: 5dc109126329457e8f01aa9357f41fa3, State: Running, Role: LEADER
I20260621 07:56:19.449949 2367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } }
I20260621 07:56:19.450294 2374 sys_catalog.cc:455] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } } }
I20260621 07:56:19.450591 2374 sys_catalog.cc:458] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:56:19.452291 2367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dc109126329457e8f01aa9357f41fa3. Latest consensus state: current_term: 1 leader_uuid: "5dc109126329457e8f01aa9357f41fa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } } }
I20260621 07:56:19.452435 2367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [sys.catalog]: This master's current role is: LEADER
W20260621 07:56:19.456750 2398 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:56:19.456848 2398 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:56:19.459434 2411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:56:19.461018 2411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260621 07:56:19.463711 2407 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:56:19.463781 2407 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:56:19.466499 2332 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d [term 1 FOLLOWER]: Refusing update from remote peer 5dc109126329457e8f01aa9357f41fa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:56:19.467685 2367 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:56:19.471997 2374 sys_catalog.cc:455] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dc109126329457e8f01aa9357f41fa3. Latest consensus state: current_term: 1 leader_uuid: "5dc109126329457e8f01aa9357f41fa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } } }
I20260621 07:56:19.472127 2374 sys_catalog.cc:458] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:56:19.475492 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.455005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.12.12.61:40065)
0621 07:56:19.455117 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.455122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.466285 (+ 11163us) server_negotiation.cc:407] Connection header received
0621 07:56:19.466489 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.466493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.466547 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.466601 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.467614 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.468398 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.469579 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.469908 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.469981 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.471939 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:19.471943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:19.471946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:19.471970 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:19.473676 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.474150 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.474154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.474156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.474234 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.474573 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.474576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.474578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.474777 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:19.474871 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:19.475198 (+ 327us) server_negotiation.cc:299] Negotiation successful
0621 07:56:19.475315 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260621 07:56:19.479624 2408 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:56:19.479707 2408 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:56:19.480103 2255 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d [term 1 FOLLOWER]: Refusing update from remote peer 5dc109126329457e8f01aa9357f41fa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:56:19.480716 2367 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:56:19.482903 2367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5dc109126329457e8f01aa9357f41fa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } } }
I20260621 07:56:19.483022 2367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:19.483589 2416 mvcc.cc:205] Tried to move back new op lower bound from 7299189061486804992 to 7299189061432651776. Current Snapshot: MvccSnapshot[applied={T|T < 7299189061486804992}]
I20260621 07:56:19.484345 2374 sys_catalog.cc:455] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5dc109126329457e8f01aa9357f41fa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } } }
I20260621 07:56:19.484465 2374 sys_catalog.cc:458] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:56:19.484748 2373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dc109126329457e8f01aa9357f41fa3. Latest consensus state: current_term: 1 leader_uuid: "5dc109126329457e8f01aa9357f41fa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } } }
I20260621 07:56:19.484865 2373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:56:19.485018 2418 mvcc.cc:205] Tried to move back new op lower bound from 7299189061486804992 to 7299189061432651776. Current Snapshot: MvccSnapshot[applied={T|T < 7299189061486804992}]
I20260621 07:56:19.490658 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.453173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39206 (local address 127.12.12.60:40523)
0621 07:56:19.453289 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.453294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.465702 (+ 12408us) server_negotiation.cc:407] Connection header received
0621 07:56:19.467160 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.467163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.467204 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.467278 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.472306 (+ 5028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.473074 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.475325 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.475661 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.475737 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.485137 (+ 9400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:19.485142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:19.485148 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:19.485181 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:19.487007 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.487582 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.487586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.487588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.487676 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.489147 (+ 1471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.489151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.489153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.489386 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:19.489532 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:19.490351 (+ 819us) server_negotiation.cc:299] Negotiation successful
0621 07:56:19.490488 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:56:19.496501 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:19.454253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.12.12.62:38159)
0621 07:56:19.454365 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.454369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.466872 (+ 12503us) server_negotiation.cc:407] Connection header received
0621 07:56:19.466921 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.466924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.466971 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.467050 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.476049 (+ 8999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.476823 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.487820 (+ 10997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:19.488213 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:19.488292 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:19.492887 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:19.492892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:19.492895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:19.492925 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:19.494659 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.495144 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.495148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.495150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.495233 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:19.495532 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:19.495535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:19.495537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:19.495747 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:19.495868 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:19.496195 (+ 327us) server_negotiation.cc:299] Negotiation successful
0621 07:56:19.496335 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 07:56:19.510902 2411 catalog_manager.cc:1347] Generated new cluster ID: 45c24baa145d4f41afd2d4961b5d868b
I20260621 07:56:19.510996 2411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:56:19.527432 2401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5dc109126329457e8f01aa9357f41fa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } } }
I20260621 07:56:19.527630 2401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:19.533901 2373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5dc109126329457e8f01aa9357f41fa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc109126329457e8f01aa9357f41fa3" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 38159 } } peers { permanent_uuid: "4b9a1a45b38a4293a33afff4b45c737d" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 40065 } } peers { permanent_uuid: "f16df8c809b64b89ac673a4b63f3507d" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 40523 } } }
I20260621 07:56:19.534051 2373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:56:19.535984 2422 mvcc.cc:205] Tried to move back new op lower bound from 7299189061486804992 to 7299189061432651776. Current Snapshot: MvccSnapshot[applied={T|T < 7299189061486804992 or (T in {7299189061486804992})}]
I20260621 07:56:19.537688 2411 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:56:19.539330 2411 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:56:19.552608 2411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dc109126329457e8f01aa9357f41fa3: Generated new TSK 0
I20260621 07:56:19.553577 2411 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:56:19.593261 2364 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:19.590639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.12.12.60:40523)
0621 07:56:19.590786 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.590791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.592150 (+ 1359us) server_negotiation.cc:407] Connection header received
0621 07:56:19.592361 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.592365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.592414 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.592468 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.593123 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39220: BlockingRecv error: recv got EOF from 127.0.0.1:39220 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260621 07:56:19.594749 2209 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:19.591529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.12.12.62:38159)
0621 07:56:19.591640 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.591645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.593757 (+ 2112us) server_negotiation.cc:407] Connection header received
0621 07:56:19.593939 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.593943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.593993 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.594043 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.594621 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50802: BlockingRecv error: recv got EOF from 127.0.0.1:50802 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260621 07:56:19.596724 2287 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:19.595227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.12.12.61:40065)
0621 07:56:19.595353 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.595358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.595820 (+ 462us) server_negotiation.cc:407] Connection header received
0621 07:56:19.595990 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.595993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.596039 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.596090 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.596605 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53884: BlockingRecv error: recv got EOF from 127.0.0.1:53884 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260621 07:56:19.601513 2287 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:19.599812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.12.12.61:40065)
0621 07:56:19.599927 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.599931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.600389 (+ 458us) server_negotiation.cc:407] Connection header received
0621 07:56:19.600626 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.600629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.600704 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.600756 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.601302 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53892: BlockingRecv error: recv got EOF from 127.0.0.1:53892 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260621 07:56:19.607162 2287 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:19.605460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.12.12.61:40065)
0621 07:56:19.605573 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.605578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.606096 (+ 518us) server_negotiation.cc:407] Connection header received
0621 07:56:19.606302 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.606306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.606372 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.606474 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.607046 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53906: BlockingRecv error: recv got EOF from 127.0.0.1:53906 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260621 07:56:19.613224 2287 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:19.611624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.12.12.61:40065)
0621 07:56:19.611730 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.611735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.612228 (+ 493us) server_negotiation.cc:407] Connection header received
0621 07:56:19.612447 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.612451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.612506 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.612562 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.613114 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53918: BlockingRecv error: recv got EOF from 127.0.0.1:53918 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260621 07:56:19.621388 2287 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:19.619697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.12.12.61:40065)
0621 07:56:19.619844 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.619849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.620376 (+ 527us) server_negotiation.cc:407] Connection header received
0621 07:56:19.620584 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.620589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.620641 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.620699 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.621251 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53934: BlockingRecv error: recv got EOF from 127.0.0.1:53934 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260621 07:56:19.633816 2287 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:19.632052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.12.12.61:40065)
0621 07:56:19.632182 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.632187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.632748 (+ 561us) server_negotiation.cc:407] Connection header received
0621 07:56:19.632984 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.632987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.633044 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.633108 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.633692 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53950: BlockingRecv error: recv got EOF from 127.0.0.1:53950 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260621 07:56:19.654198 2287 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:19.652494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.12.12.61:40065)
0621 07:56:19.652615 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.652619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.653161 (+ 542us) server_negotiation.cc:407] Connection header received
0621 07:56:19.653414 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.653418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.653470 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.653523 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.654077 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53962: BlockingRecv error: recv got EOF from 127.0.0.1:53962 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260621 07:56:19.690261 2287 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:19.688958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.12.12.61:40065)
0621 07:56:19.689130 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.689136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.689312 (+ 176us) server_negotiation.cc:407] Connection header received
0621 07:56:19.689504 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.689507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.689557 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.689650 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.690149 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53974: BlockingRecv error: recv got EOF from 127.0.0.1:53974 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260621 07:56:19.757743 2287 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:19.756530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.12.12.61:40065)
0621 07:56:19.756688 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.756693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.756909 (+ 216us) server_negotiation.cc:407] Connection header received
0621 07:56:19.757012 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.757015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.757067 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.757145 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.757613 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53984: BlockingRecv error: recv got EOF from 127.0.0.1:53984 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260621 07:56:19.889387 2287 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:19.888105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.12.12.61:40065)
0621 07:56:19.888279 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:19.888284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:19.888311 (+ 27us) server_negotiation.cc:407] Connection header received
0621 07:56:19.888497 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:19.888500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:19.888553 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:19.888635 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:19.889249 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53990: BlockingRecv error: recv got EOF from 127.0.0.1:53990 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260621 07:56:20.149052 2287 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:20.147678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.12.12.61:40065)
0621 07:56:20.147860 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:20.147865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:20.148026 (+ 161us) server_negotiation.cc:407] Connection header received
0621 07:56:20.148146 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:20.148149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:20.148200 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:20.148264 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:20.148929 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53992: BlockingRecv error: recv got EOF from 127.0.0.1:53992 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:56:20.465134 2407 catalog_manager.cc:1259] Loaded cluster ID: 45c24baa145d4f41afd2d4961b5d868b
I20260621 07:56:20.465242 2407 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d: loading cluster ID for follower catalog manager: success
I20260621 07:56:20.466876 2407 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d: acquiring CA information for follower catalog manager: success
I20260621 07:56:20.467994 2407 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f16df8c809b64b89ac673a4b63f3507d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:56:20.481061 2408 catalog_manager.cc:1259] Loaded cluster ID: 45c24baa145d4f41afd2d4961b5d868b
I20260621 07:56:20.481148 2408 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d: loading cluster ID for follower catalog manager: success
I20260621 07:56:20.482969 2408 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d: acquiring CA information for follower catalog manager: success
I20260621 07:56:20.484118 2408 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b9a1a45b38a4293a33afff4b45c737d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:56:20.667135 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:20.663201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.12.12.61:40065)
0621 07:56:20.663448 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:20.663452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:20.663476 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:56:20.663610 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:20.663613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:20.663734 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:20.663848 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:56:20.664186 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:20.664724 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:20.665659 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:20.665909 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:20.666001 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:20.666303 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:56:20.666790 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:56:20.666888 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:20.666959 (+ 71us) server_negotiation.cc:299] Negotiation successful
0621 07:56:20.667009 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":96,"threads_started":1}
I20260621 07:56:20.673883 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:20.670452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.12.12.60:40523)
0621 07:56:20.670693 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:20.670697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:20.670721 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:56:20.670799 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:20.670801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:20.670910 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:20.671016 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:56:20.671215 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:20.671716 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:20.672514 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:20.672755 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:20.672836 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:20.673073 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:56:20.673491 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:56:20.673582 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:20.673630 (+ 48us) server_negotiation.cc:299] Negotiation successful
0621 07:56:20.673675 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":101,"threads_started":1}
I20260621 07:56:20.681406 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:20.677102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54004 (local address 127.12.12.61:40065)
0621 07:56:20.677671 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:20.677675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:20.677687 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:56:20.677733 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:20.677736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:20.677781 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:20.677867 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:56:20.678117 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:20.678798 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:20.680085 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:20.680337 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:20.680456 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:20.680722 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:56:20.680813 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:56:20.680895 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:20.681200 (+ 305us) server_negotiation.cc:299] Negotiation successful
0621 07:56:20.681261 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487}
I20260621 07:56:20.681582 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:20.677212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.12.12.60:40523)
0621 07:56:20.677565 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:20.677570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:20.677667 (+ 97us) server_negotiation.cc:407] Connection header received
0621 07:56:20.677777 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:20.677780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:20.677828 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:20.677900 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:56:20.678182 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:20.678976 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:20.680096 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:20.680418 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:20.680505 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:20.680984 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:56:20.681060 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:56:20.681130 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:20.681388 (+ 258us) server_negotiation.cc:299] Negotiation successful
0621 07:56:20.681448 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277}
I20260621 07:56:20.681957 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:20.677337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.12.12.62:38159)
0621 07:56:20.677565 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:20.677570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:20.677585 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:56:20.677683 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:20.677687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:20.677853 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:20.677977 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:56:20.678362 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:20.679168 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:20.680024 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:20.680374 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:20.680484 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:20.681077 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:56:20.681496 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:56:20.681582 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:20.681708 (+ 126us) server_negotiation.cc:299] Negotiation successful
0621 07:56:20.681772 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":92,"threads_started":1}
I20260621 07:56:20.685285 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 2138
I20260621 07:56:20.691849 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 2213
I20260621 07:56:20.697602 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 2288
2026-06-21T07:56:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3136 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:56:20 dist-test-slave-lbpt krb5kdc[2508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:56:20 dist-test-slave-lbpt krb5kdc[2508](info): set up 2 sockets
Jun 21 07:56:20 dist-test-slave-lbpt krb5kdc[2508](info): commencing operation
krb5kdc: starting...
W20260621 07:56:21.734787 12336 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/test-user.keytab.
Jun 21 07:56:21 dist-test-slave-lbpt krb5kdc[2508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:56:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:56:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
Entry for principal kudu/127.12.12.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab.
I20260621 07:56:21.853345 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:41045
--webserver_interface=127.12.12.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:44463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:41045,127.12.12.61:38265,127.12.12.60:41373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:21.965714 2521 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:56:21.966073 2521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:21.966135 2521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:21.966226 2521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:21.970316 2521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:21.970402 2521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:21.970456 2521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:21.970492 2521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:21.970527 2521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:21.975863 2521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:44463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:41045,127.12.12.61:38265,127.12.12.60:41373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.62.keytab
--principal=kudu/127.12.12.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.62:41045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:21.976963 2521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:21.977993 2521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:21.986625 2526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:21.986650 2527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:21.986723 2521 server_base.cc:1034] running on GCE node
W20260621 07:56:21.986650 2529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:21.987498 2521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:21.988629 2521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:21.989825 2521 hybrid_clock.cc:648] HybridClock initialized: now 1782028581989814 us; error 69 us; skew 500 ppm
Jun 21 07:56:21 dist-test-slave-lbpt krb5kdc[2508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028581, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:21.993021 2521 init.cc:377] Logged in from keytab as kudu/127.12.12.62@KRBTEST.COM (short username kudu)
I20260621 07:56:21.994359 2521 webserver.cc:460] Webserver started at http://127.12.12.62:41921/ using document root <none> and password file <none>
I20260621 07:56:21.994956 2521 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:21.995050 2521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:21.995280 2521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:21.997892 2521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/data/instance:
uuid: "29cca06f2b8e4d1ab368dd88c8eefd20"
format_stamp: "Formatted at 2026-06-21 07:56:21 on dist-test-slave-lbpt"
server_key: "0468f64878777d0bffaa3aad5beb9266"
server_key_iv: "27b5c8dc45090faa88b986ade3bf4d23"
server_key_version: "encryptionkey@0"
I20260621 07:56:21.998695 2521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/wal/instance:
uuid: "29cca06f2b8e4d1ab368dd88c8eefd20"
format_stamp: "Formatted at 2026-06-21 07:56:21 on dist-test-slave-lbpt"
server_key: "0468f64878777d0bffaa3aad5beb9266"
server_key_iv: "27b5c8dc45090faa88b986ade3bf4d23"
server_key_version: "encryptionkey@0"
I20260621 07:56:22.002835 2521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 07:56:22.005334 2536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:22.006729 2521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:56:22.006840 2521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/wal
uuid: "29cca06f2b8e4d1ab368dd88c8eefd20"
format_stamp: "Formatted at 2026-06-21 07:56:21 on dist-test-slave-lbpt"
server_key: "0468f64878777d0bffaa3aad5beb9266"
server_key_iv: "27b5c8dc45090faa88b986ade3bf4d23"
server_key_version: "encryptionkey@0"
I20260621 07:56:22.006959 2521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:22.026226 2521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:22.029462 2521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:22.029696 2521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:22.037117 2521 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.62:41045
I20260621 07:56:22.037405 2588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.62:41045 every 8 connection(s)
I20260621 07:56:22.038182 2521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/data/info.pb
I20260621 07:56:22.039889 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 2521
I20260621 07:56:22.040021 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-0/wal/instance
I20260621 07:56:22.040306 12336 external_mini_cluster.cc:1428] Setting key 2e42dc62525d5721d580108771c1b84c
I20260621 07:56:22.041065 2589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:22.044763 2589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.12.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.61@KRBTEST.COM" created.
Jun 21 07:56:22 dist-test-slave-lbpt krb5kdc[2508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028581, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:56:22.061100 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.047165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:41045 (local address 127.0.0.1:58898)
0621 07:56:22.047735 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.047792 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:22.048657 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:22.049362 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:22.049368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:22.049701 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:22.049959 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.049972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.051478 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.051485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.052401 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.052411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.053274 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.053281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.053499 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.054164 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:22.054189 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:22.055929 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:22.057911 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.057921 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.057931 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.058167 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.058536 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.058541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.058546 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.058670 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.059340 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:22.059347 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:22.059494 (+ 147us) client_negotiation.cc:769] Sending connection context
0621 07:56:22.059666 (+ 172us) client_negotiation.cc:240] Negotiation successful
0621 07:56:22.060064 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":153,"thread_start_us":140,"threads_started":1}
I20260621 07:56:22.061100 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.047165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58898 (local address 127.12.12.62:41045)
0621 07:56:22.047719 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.047736 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.047914 (+ 178us) server_negotiation.cc:407] Connection header received
0621 07:56:22.048949 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.048953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.049142 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.049330 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.050163 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.051344 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.052537 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.053110 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.053249 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.056084 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:22.056091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:22.056109 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:22.056129 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:22.057783 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.058281 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.058287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.058289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.058396 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.058771 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.058777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.058779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.059192 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:22.059373 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:22.059779 (+ 406us) server_negotiation.cc:299] Negotiation successful
0621 07:56:22.060012 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":437,"spinlock_wait_cycles":22272,"thread_start_us":169,"threads_started":1}
I20260621 07:56:22.063812 2589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:56:22.064574 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.064259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:38265 (local address 127.0.0.1:44738)
0621 07:56:22.064388 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.064461 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.61.keytab.
Entry for principal kudu/127.12.12.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.61.keytab.
W20260621 07:56:22.064817 2538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.12.61:38265: connect: Connection refused (error 111)
I20260621 07:56:22.065738 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.065540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:38265 (local address 127.0.0.1:44750)
0621 07:56:22.065642 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.065688 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 07:56:22.065846 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.61:38265
--webserver_interface=127.12.12.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.61.keytab
--principal=kudu/127.12.12.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:44463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:41045,127.12.12.61:38265,127.12.12.60:41373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:22.065953 2589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.61:38265: Network error: Client connection negotiation failed: client connection to 127.12.12.61:38265: connect: Connection refused (error 111)
I20260621 07:56:22.115226 2589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } attempt: 1
I20260621 07:56:22.116205 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.115849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:38265 (local address 127.0.0.1:44766)
0621 07:56:22.116038 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.116112 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260621 07:56:22.116971 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.116741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:38265 (local address 127.0.0.1:44772)
0621 07:56:22.116851 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.116904 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 07:56:22.117151 2589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.61:38265: Network error: Client connection negotiation failed: client connection to 127.12.12.61:38265: connect: Connection refused (error 111)
I20260621 07:56:22.185369 2589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } attempt: 2
I20260621 07:56:22.186314 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.185972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:38265 (local address 127.0.0.1:44780)
0621 07:56:22.186140 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.186212 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260621 07:56:22.186298 2596 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:56:22.186713 2596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:22.186820 2596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:22.186877 2596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260621 07:56:22.187187 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.186962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:38265 (local address 127.0.0.1:44782)
0621 07:56:22.187074 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.187126 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.61:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260621 07:56:22.187338 2589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.61:38265: Network error: Client connection negotiation failed: client connection to 127.12.12.61:38265: connect: Connection refused (error 111)
W20260621 07:56:22.191287 2596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:22.191375 2596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:22.191423 2596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:22.191459 2596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:22.191501 2596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:22.197054 2596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:44463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:41045,127.12.12.61:38265,127.12.12.60:41373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.61.keytab
--principal=kudu/127.12.12.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.61:38265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.12.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:22.198191 2596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:22.199348 2596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:56:22.208441 2596 server_base.cc:1034] running on GCE node
W20260621 07:56:22.208343 2602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:22.208374 2604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:22.208364 2601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:22.209278 2596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:22.210573 2596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:22.211879 2596 hybrid_clock.cc:648] HybridClock initialized: now 1782028582211879 us; error 41 us; skew 500 ppm
Jun 21 07:56:22 dist-test-slave-lbpt krb5kdc[2508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028582, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:22.215020 2596 init.cc:377] Logged in from keytab as kudu/127.12.12.61@KRBTEST.COM (short username kudu)
I20260621 07:56:22.216024 2596 webserver.cc:460] Webserver started at http://127.12.12.61:33405/ using document root <none> and password file <none>
I20260621 07:56:22.216576 2596 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:22.216637 2596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:22.216820 2596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:22.219084 2596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/data/instance:
uuid: "12b183890e6b47fa98aa35ec83d2d319"
format_stamp: "Formatted at 2026-06-21 07:56:22 on dist-test-slave-lbpt"
server_key: "615a693408d3ad376f71380da6337cc5"
server_key_iv: "3cd7819a7305f5273464de90deba4e93"
server_key_version: "encryptionkey@0"
I20260621 07:56:22.219709 2596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/wal/instance:
uuid: "12b183890e6b47fa98aa35ec83d2d319"
format_stamp: "Formatted at 2026-06-21 07:56:22 on dist-test-slave-lbpt"
server_key: "615a693408d3ad376f71380da6337cc5"
server_key_iv: "3cd7819a7305f5273464de90deba4e93"
server_key_version: "encryptionkey@0"
I20260621 07:56:22.223634 2596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260621 07:56:22.226172 2611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:22.227574 2596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:56:22.227733 2596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/wal
uuid: "12b183890e6b47fa98aa35ec83d2d319"
format_stamp: "Formatted at 2026-06-21 07:56:22 on dist-test-slave-lbpt"
server_key: "615a693408d3ad376f71380da6337cc5"
server_key_iv: "3cd7819a7305f5273464de90deba4e93"
server_key_version: "encryptionkey@0"
I20260621 07:56:22.227878 2596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:22.252884 2596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:22.256624 2596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:22.256865 2596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:22.264673 2596 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.61:38265
I20260621 07:56:22.264988 2663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.61:38265 every 8 connection(s)
I20260621 07:56:22.265739 2596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/data/info.pb
I20260621 07:56:22.268884 2664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:22.272737 2664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:56:22.273326 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 2596
I20260621 07:56:22.273453 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-1/wal/instance
I20260621 07:56:22.273754 12336 external_mini_cluster.cc:1428] Setting key 4b70431e22f9871d455b12278c1956ef
I20260621 07:56:22.278481 2589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } attempt: 3
WARNING: no policy specified for kudu/127.12.12.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.60@KRBTEST.COM" created.
Jun 21 07:56:22 dist-test-slave-lbpt krb5kdc[2508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028582, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.61@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
Jun 21 07:56:22 dist-test-slave-lbpt krb5kdc[2508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028581, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for kudu/127.12.12.61@KRBTEST.COM
I20260621 07:56:22.291437 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.278962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:38265 (local address 127.0.0.1:44788)
0621 07:56:22.279172 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.279187 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:22.279256 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:22.279988 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:22.279993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:22.280202 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:22.280350 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.280358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.281664 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.281668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.282394 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.282405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.283321 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.283327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.283449 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.283966 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:22.283986 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:22.286341 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:22.289268 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.289274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.289279 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.289558 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.289953 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.289956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.289958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.290049 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.290736 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:22.290741 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:22.290883 (+ 142us) client_negotiation.cc:769] Sending connection context
0621 07:56:22.290989 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 07:56:22.291179 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":49}
I20260621 07:56:22.291985 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.275637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:41045 (local address 127.0.0.1:58912)
0621 07:56:22.276236 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.276282 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:22.277222 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:22.277852 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:22.277862 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:22.278281 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:22.279009 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.279031 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.280116 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.280118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.281392 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.281403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.282232 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.282236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.282481 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.283166 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:22.283183 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:22.285985 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:22.288256 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.288267 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.288281 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.288670 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.289039 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.289043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.289045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.289242 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.289698 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:22.289705 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:22.290063 (+ 358us) client_negotiation.cc:769] Sending connection context
0621 07:56:22.290306 (+ 243us) client_negotiation.cc:240] Negotiation successful
0621 07:56:22.290763 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":174,"threads_started":1}
I20260621 07:56:22.292080 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.275120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58912 (local address 127.12.12.62:41045)
0621 07:56:22.275310 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.275318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.276436 (+ 1118us) server_negotiation.cc:407] Connection header received
0621 07:56:22.277462 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.277465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.277527 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.277645 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.279172 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.279959 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.281531 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.282061 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.282171 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.286235 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:22.286242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:22.286249 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:22.286311 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:22.287991 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.288809 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.288816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.288821 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.288922 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.289326 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.289330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.289331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.289595 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:22.289700 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:22.291061 (+ 1361us) server_negotiation.cc:299] Negotiation successful
0621 07:56:22.291262 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":65}
I20260621 07:56:22.291987 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.279105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44788 (local address 127.12.12.61:38265)
0621 07:56:22.279514 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.279529 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.279551 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:56:22.279589 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.279595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.279788 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.279957 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.280482 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.281559 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.282545 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.283152 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.283415 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.286511 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:22.286518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:22.286532 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:22.286893 (+ 361us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:22.289152 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.289712 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.289723 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.289727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.289847 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.290159 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.290163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.290167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.290568 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:22.291446 (+ 878us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:22.291578 (+ 132us) server_negotiation.cc:299] Negotiation successful
0621 07:56:22.291752 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":173,"threads_started":1}
I20260621 07:56:22.293603 2664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:56:22.294445 2589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:56:22.295102 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.294896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:41373 (local address 127.0.0.1:42188)
0621 07:56:22.294989 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.295042 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260621 07:56:22.295938 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.295649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:41373 (local address 127.0.0.1:42196)
0621 07:56:22.295814 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.295865 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260621 07:56:22.296131 2589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:41373: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
Jun 21 07:56:22 dist-test-slave-lbpt krb5kdc[2508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028582, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.61@KRBTEST.COM for kudu/127.12.12.61@KRBTEST.COM
I20260621 07:56:22.303292 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.294455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44798 (local address 127.12.12.61:38265)
0621 07:56:22.294583 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.294587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.294601 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:56:22.294710 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.294715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.294777 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.294887 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.295625 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.296407 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.297238 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.297490 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.297598 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.299927 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:22.299931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:22.299933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:22.299955 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:22.301414 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.301938 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.301942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.301944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.302026 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.302338 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.302342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.302343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.302578 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:22.302687 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:22.302919 (+ 232us) server_negotiation.cc:299] Negotiation successful
0621 07:56:22.303081 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":51}
I20260621 07:56:22.303292 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.294196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:38265 (local address 127.0.0.1:44798)
0621 07:56:22.294499 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.294514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:22.294611 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:22.294919 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:22.295255 (+ 336us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:22.295405 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:22.295509 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.295514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.296513 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.296520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.297124 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.297132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.297596 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.297599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.297734 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.298198 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:22.298210 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:22.299787 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:22.301519 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.301526 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.301528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.301813 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.302117 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.302122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.302124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.302207 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.302673 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:22.302676 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:22.302762 (+ 86us) client_negotiation.cc:769] Sending connection context
0621 07:56:22.302877 (+ 115us) client_negotiation.cc:240] Negotiation successful
0621 07:56:22.303137 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":139}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.60.keytab.
Entry for principal kudu/127.12.12.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.60.keytab.
I20260621 07:56:22.304452 2664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:56:22.305506 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.305093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:41373 (local address 127.0.0.1:42198)
0621 07:56:22.305268 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.305410 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260621 07:56:22.305517 12336 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
/tmp/dist-test-taskzeuNYm/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.60:41373
--webserver_interface=127.12.12.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.60.keytab
--principal=kudu/127.12.12.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.12.20:44463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.62:41045,127.12.12.61:38265,127.12.12.60:41373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:56:22.306169 2612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
I20260621 07:56:22.307199 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.306927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:41373 (local address 127.0.0.1:42212)
0621 07:56:22.307043 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.307106 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260621 07:56:22.307425 2664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:41373: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
I20260621 07:56:22.327315 2589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } attempt: 1
I20260621 07:56:22.328280 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.327976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:41373 (local address 127.0.0.1:42222)
0621 07:56:22.328132 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.328202 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260621 07:56:22.329058 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.328813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:41373 (local address 127.0.0.1:42234)
0621 07:56:22.328952 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.329001 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260621 07:56:22.329228 2589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:41373: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
I20260621 07:56:22.356729 2664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } attempt: 1
I20260621 07:56:22.357638 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.357338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:41373 (local address 127.0.0.1:42240)
0621 07:56:22.357488 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.357558 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260621 07:56:22.358564 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.358207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:41373 (local address 127.0.0.1:42244)
0621 07:56:22.358382 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.358505 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260621 07:56:22.358780 2664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:41373: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
I20260621 07:56:22.404404 2589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } attempt: 2
I20260621 07:56:22.405227 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.404923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:41373 (local address 127.0.0.1:42254)
0621 07:56:22.405083 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.405149 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 07:56:22.405895 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.405685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:41373 (local address 127.0.0.1:42268)
0621 07:56:22.405793 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.405840 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260621 07:56:22.406018 2589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:41373: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
W20260621 07:56:22.420416 2671 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:56:22.420687 2671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:56:22.420760 2671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:56:22.420840 2671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:56:22.424909 2671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:56:22.424971 2671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:56:22.424991 2671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:56:22.425007 2671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:56:22.425024 2671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:56:22.426959 2664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } attempt: 2
I20260621 07:56:22.427806 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.427507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:41373 (local address 127.0.0.1:59464)
0621 07:56:22.427670 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.427731 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260621 07:56:22.428508 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.428301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:41373 (local address 127.0.0.1:59476)
0621 07:56:22.428404 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.428456 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260621 07:56:22.428618 2664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.12.60:41373: Network error: Client connection negotiation failed: client connection to 127.12.12.60:41373: connect: Connection refused (error 111)
I20260621 07:56:22.430960 2671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.20:44463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.62:41045,127.12.12.61:38265,127.12.12.60:41373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/krb5kdc/kudu_127.12.12.60.keytab
--principal=kudu/127.12.12.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.60:41373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.12.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:56:22.432130 2671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:56:22.433297 2671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:56:22.442301 2677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:56:22.442307 2676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:22.442401 2671 server_base.cc:1034] running on GCE node
W20260621 07:56:22.442310 2679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:56:22.443312 2671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:56:22.444511 2671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:56:22.445758 2671 hybrid_clock.cc:648] HybridClock initialized: now 1782028582445690 us; error 52 us; skew 500 ppm
Jun 21 07:56:22 dist-test-slave-lbpt krb5kdc[2508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028582, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:56:22.448987 2671 init.cc:377] Logged in from keytab as kudu/127.12.12.60@KRBTEST.COM (short username kudu)
I20260621 07:56:22.450047 2671 webserver.cc:460] Webserver started at http://127.12.12.60:34377/ using document root <none> and password file <none>
I20260621 07:56:22.450701 2671 fs_manager.cc:359] Metadata directory not provided
I20260621 07:56:22.450820 2671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:56:22.451059 2671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:56:22.453209 2671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/data/instance:
uuid: "c93987f97b524c46b79ce0f7a04cdad0"
format_stamp: "Formatted at 2026-06-21 07:56:22 on dist-test-slave-lbpt"
server_key: "15228cfaf1f8911f90bf98ef87c9863a"
server_key_iv: "ed47c3a77e7e8b7f0a63a47918f16f6d"
server_key_version: "encryptionkey@0"
I20260621 07:56:22.453806 2671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/wal/instance:
uuid: "c93987f97b524c46b79ce0f7a04cdad0"
format_stamp: "Formatted at 2026-06-21 07:56:22 on dist-test-slave-lbpt"
server_key: "15228cfaf1f8911f90bf98ef87c9863a"
server_key_iv: "ed47c3a77e7e8b7f0a63a47918f16f6d"
server_key_version: "encryptionkey@0"
I20260621 07:56:22.457823 2671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260621 07:56:22.460569 2686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:56:22.461966 2671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:56:22.462109 2671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/data,/tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/wal
uuid: "c93987f97b524c46b79ce0f7a04cdad0"
format_stamp: "Formatted at 2026-06-21 07:56:22 on dist-test-slave-lbpt"
server_key: "15228cfaf1f8911f90bf98ef87c9863a"
server_key_iv: "ed47c3a77e7e8b7f0a63a47918f16f6d"
server_key_version: "encryptionkey@0"
I20260621 07:56:22.462234 2671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:56:22.479256 2671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:56:22.482685 2671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:56:22.482906 2671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:56:22.490016 2671 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.60:41373
I20260621 07:56:22.490309 2738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.12.60:41373 every 8 connection(s)
I20260621 07:56:22.491096 2671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/data/info.pb
I20260621 07:56:22.492354 12336 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu as pid 2671
I20260621 07:56:22.492491 12336 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzeuNYm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028299065733-12336-0/minicluster-data/master-2/wal/instance
I20260621 07:56:22.492799 12336 external_mini_cluster.cc:1428] Setting key 3f08a6d0dbd2bb35ba95b2c5ade3ac10
I20260621 07:56:22.494630 2739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:56:22.499300 2739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:56:22 dist-test-slave-lbpt krb5kdc[2508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.60@KRBTEST.COM
I20260621 07:56:22.505218 2589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } attempt: 3
Jun 21 07:56:22 dist-test-slave-lbpt krb5kdc[2508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028582, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.60@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
I20260621 07:56:22.509497 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.494224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.12.12.60:41373)
0621 07:56:22.494722 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.494742 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.494784 (+ 42us) server_negotiation.cc:407] Connection header received
0621 07:56:22.495651 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.495673 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.496093 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.496467 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.497202 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.498737 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.499596 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.500248 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.500400 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.503636 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:22.503645 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:22.503657 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:22.503690 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:22.506073 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.506631 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.506637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.506644 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.506798 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.507207 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.507210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.507212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.507701 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:22.507929 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:22.508171 (+ 242us) server_negotiation.cc:299] Negotiation successful
0621 07:56:22.508597 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":196,"threads_started":1}
Jun 21 07:56:22 dist-test-slave-lbpt krb5kdc[2508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028581, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.62@KRBTEST.COM for kudu/127.12.12.60@KRBTEST.COM
I20260621 07:56:22.511442 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.501249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.12.12.62:41045)
0621 07:56:22.501404 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.501408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.501777 (+ 369us) server_negotiation.cc:407] Connection header received
0621 07:56:22.501929 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.501931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.501975 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.502043 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.502969 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.503527 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.504396 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.504756 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.504886 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.507557 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:22.507561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:22.507563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:22.507589 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:22.509333 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.509897 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.509901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.509902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.509980 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.510350 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.510354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.510356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.510603 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:22.510720 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:22.511132 (+ 412us) server_negotiation.cc:299] Negotiation successful
0621 07:56:22.511292 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 07:56:22.512836 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.501170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.62:41045 (local address 127.0.0.1:52974)
0621 07:56:22.501618 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.501681 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:22.501824 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:22.502127 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:22.502144 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:22.502698 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:22.502843 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.502849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.503638 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.503640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.504273 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.504282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.504868 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.504871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.504963 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.505486 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:22.505503 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:22.507449 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:22.509451 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.509459 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.509464 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.509750 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.510126 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.510128 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.510130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.510232 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.510731 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:22.510739 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:22.510847 (+ 108us) client_negotiation.cc:769] Sending connection context
0621 07:56:22.511633 (+ 786us) client_negotiation.cc:240] Negotiation successful
0621 07:56:22.512072 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":157,"threads_started":1}
I20260621 07:56:22.514135 2739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:56:22.515218 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.505636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:41373 (local address 127.0.0.1:59498)
0621 07:56:22.505841 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.505855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:22.505946 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:22.506336 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:22.506339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:22.506557 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:22.506661 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.506667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.507460 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.507466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.508304 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.508317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.509193 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.509198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.509302 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.509737 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:22.509750 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:22.511254 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:22.513651 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.513654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.513656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.513924 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.514217 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.514221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.514222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.514314 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.514757 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:22.514760 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:22.514847 (+ 87us) client_negotiation.cc:769] Sending connection context
0621 07:56:22.514931 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 07:56:22.515062 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260621 07:56:22.515856 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.505711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.12.12.60:41373)
0621 07:56:22.506088 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.506097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.506111 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:56:22.506144 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.506147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.506197 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.506304 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.506775 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.507355 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.508678 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.509048 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.509512 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.511389 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:22.511393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:22.511396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:22.511418 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:22.513542 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.514018 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.514021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.514023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.514109 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.514451 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.514454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.514456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.514647 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:22.515488 (+ 841us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:22.515563 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 07:56:22.515683 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":110,"threads_started":1}
Jun 21 07:56:22 dist-test-slave-lbpt krb5kdc[2508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.62@KRBTEST.COM
Jun 21 07:56:22 dist-test-slave-lbpt krb5kdc[2508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028582, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.60@KRBTEST.COM for kudu/127.12.12.61@KRBTEST.COM
I20260621 07:56:22.519611 2589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20: Bootstrap starting.
I20260621 07:56:22.519747 2664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } attempt: 3
I20260621 07:56:22.522694 2589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:22.523065 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.512701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52990 (local address 127.12.12.62:41045)
0621 07:56:22.512845 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.512852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.512867 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:56:22.512905 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.512909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.512962 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.513040 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.513896 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.514630 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.515718 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.516054 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.516152 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.519033 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:22.519037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:22.519039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:22.519070 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:22.521208 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.521715 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.521718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.521720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.521809 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.522118 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.522121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.522123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.522335 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:22.522553 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:22.522770 (+ 217us) server_negotiation.cc:299] Negotiation successful
0621 07:56:22.522887 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 07:56:22.523644 2589 log.cc:824] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:22.526021 2589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20: No bootstrap required, opened a new log
Jun 21 07:56:22 dist-test-slave-lbpt krb5kdc[2508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028582, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.61@KRBTEST.COM for kudu/127.12.12.60@KRBTEST.COM
I20260621 07:56:22.527228 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.514755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.12.12.61:38265)
0621 07:56:22.514883 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.514887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.514901 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:56:22.515002 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.515005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.515049 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.515143 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.515549 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.516268 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.517183 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.517446 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.517553 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.519880 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:22.519883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:22.519886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:22.519912 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:22.521816 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.525241 (+ 3425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.525246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.525249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.525356 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.526156 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.526160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.526161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.526403 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:22.526500 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:22.526919 (+ 419us) server_negotiation.cc:299] Negotiation successful
0621 07:56:22.527039 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46,"spinlock_wait_cycles":49536}
I20260621 07:56:22.527822 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.514664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.61:38265 (local address 127.0.0.1:37544)
0621 07:56:22.514823 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.514836 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:22.514921 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:22.515154 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:22.515157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:22.515338 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:22.515449 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.515455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.516377 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.516383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.517059 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.517068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.517565 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.517568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.517662 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.518115 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:22.518128 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:22.519724 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:22.524313 (+ 4589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.524315 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.524317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.524600 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.525935 (+ 1335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.525938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.525941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.526045 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.526581 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:22.526587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:22.526692 (+ 105us) client_negotiation.cc:769] Sending connection context
0621 07:56:22.527445 (+ 753us) client_negotiation.cc:240] Negotiation successful
0621 07:56:22.527619 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260621 07:56:22.528605 2739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:56:22 dist-test-slave-lbpt krb5kdc[2508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.61@KRBTEST.COM
I20260621 07:56:22.530930 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.520176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:41373 (local address 127.0.0.1:59500)
0621 07:56:22.520435 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.520450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:22.520523 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:22.520987 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:22.520993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:22.521935 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:22.522042 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.522052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.523021 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.523025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.523700 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.523712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.524734 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.524737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.524837 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.525441 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:22.525451 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:22.526892 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:22.529300 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.529304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.529306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.529574 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.529886 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.529889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.529892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.529985 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.530435 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:22.530438 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:22.530531 (+ 93us) client_negotiation.cc:769] Sending connection context
0621 07:56:22.530612 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 07:56:22.530763 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260621 07:56:22.531634 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.520651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.12.12.60:41373)
0621 07:56:22.520756 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.520760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.520772 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:56:22.520805 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.520807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.520846 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.521173 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.522170 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.522893 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.523837 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.524183 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.525083 (+ 900us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.527039 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:22.527042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:22.527045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:22.527068 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:22.529180 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.529690 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.529693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.529695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.529776 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.530100 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.530102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.530104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.530307 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:22.531300 (+ 993us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:22.531368 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 07:56:22.531483 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:56:22.532220 2589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } }
I20260621 07:56:22.532533 2589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29cca06f2b8e4d1ab368dd88c8eefd20, State: Initialized, Role: FOLLOWER
I20260621 07:56:22.532932 2589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } }
Jun 21 07:56:22 dist-test-slave-lbpt krb5kdc[2508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028582, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.60@KRBTEST.COM for kudu/127.12.12.60@KRBTEST.COM
I20260621 07:56:22.534746 2589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:22.536343 2664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319: Bootstrap starting.
I20260621 07:56:22.536960 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.524455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.12.12.61:38265)
0621 07:56:22.524818 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.524827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.524842 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:56:22.524924 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.524927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.524970 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.525056 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.525901 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.526722 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.527792 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.528150 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.528248 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.531563 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:22.531567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:22.531570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:22.531593 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:22.533791 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.534297 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.534300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.534302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.534385 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.534759 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.534765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.534769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.534979 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:22.535162 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:22.535816 (+ 654us) server_negotiation.cc:299] Negotiation successful
0621 07:56:22.536781 (+ 965us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
I20260621 07:56:22.537388 2747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } } }
I20260621 07:56:22.537509 2747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:56:22.539109 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.529351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.12.12.60:41373)
0621 07:56:22.529496 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.529502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.529545 (+ 43us) server_negotiation.cc:407] Connection header received
0621 07:56:22.529691 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.529693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.529731 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.529780 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.530283 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.531055 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.532158 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.532503 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.532627 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.534967 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:22.534971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:22.534973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:22.534995 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:22.536653 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.537787 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.537790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.537792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.537880 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.538218 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.538222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.538224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.538494 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:22.538588 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:22.538843 (+ 255us) server_negotiation.cc:299] Negotiation successful
0621 07:56:22.538968 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:56:22.539494 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.529151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.12.60:41373 (local address 127.0.0.1:59516)
0621 07:56:22.529480 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0621 07:56:22.529497 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:56:22.529600 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:56:22.529910 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:56:22.529913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:56:22.530083 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:56:22.530179 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.530186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.531180 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.531183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.531993 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:56:22.532001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.532769 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.532776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:56:22.532874 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.533327 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:56:22.533346 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:56:22.534784 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:56:22.537233 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.537236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.537238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.537483 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.538006 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:56:22.538009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:56:22.538010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:56:22.538102 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:56:22.538620 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:56:22.538623 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:56:22.538697 (+ 74us) client_negotiation.cc:769] Sending connection context
0621 07:56:22.539240 (+ 543us) client_negotiation.cc:240] Negotiation successful
0621 07:56:22.539365 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256}
W20260621 07:56:22.540396 2758 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:56:22.540488 2758 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:56:22.539849 2521 master_runner.cc:186] Error getting master registration for 127.12.12.61:38265: OK, instance_id { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" instance_seqno: 1782028582262614 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 07:56:22.544212 2521 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 07:56:22.545127 2739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0: Bootstrap starting.
I20260621 07:56:22.545297 12336 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:56:22.546633 2563 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52990
I20260621 07:56:22.546854 2563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 07:56:22.546968 2563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:22.547485 2739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:22.548126 2739 log.cc:824] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:22.548235 2563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } }
I20260621 07:56:22.549238 2563 leader_election.cc:288] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [CANDIDATE]: Term 1 election: Requested vote from peers 12b183890e6b47fa98aa35ec83d2d319 (127.12.12.61:38265), c93987f97b524c46b79ce0f7a04cdad0 (127.12.12.60:41373)
I20260621 07:56:22.550398 2664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319: Neither blocks nor log segments found. Creating new log.
I20260621 07:56:22.551288 2664 log.cc:824] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319: Log is configured to *not* fsync() on all Append() calls
I20260621 07:56:22.553112 2739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0: No bootstrap required, opened a new log
I20260621 07:56:22.557337 2739 raft_consensus.cc:348] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } }
I20260621 07:56:22.557578 2739 raft_consensus.cc:729] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c93987f97b524c46b79ce0f7a04cdad0, State: Initialized, Role: FOLLOWER
I20260621 07:56:22.557938 2739 consensus_queue.cc:260] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } }
I20260621 07:56:22.558584 2767 sys_catalog.cc:455] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } } }
I20260621 07:56:22.558869 2767 sys_catalog.cc:458] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:56:22.559374 2739 sys_catalog.cc:564] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:22.559402 2713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c93987f97b524c46b79ce0f7a04cdad0"
I20260621 07:56:22.559592 2713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:22.561110 2713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29cca06f2b8e4d1ab368dd88c8eefd20 in term 1.
I20260621 07:56:22.561708 2537 leader_election.cc:302] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29cca06f2b8e4d1ab368dd88c8eefd20, c93987f97b524c46b79ce0f7a04cdad0; no voters:
I20260621 07:56:22.562058 2747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:56:22.562258 2747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [term 1 LEADER]: Becoming Leader. State: Replica: 29cca06f2b8e4d1ab368dd88c8eefd20, State: Running, Role: LEADER
I20260621 07:56:22.562618 2747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } }
I20260621 07:56:22.563344 2664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319: No bootstrap required, opened a new log
I20260621 07:56:22.564433 2747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29cca06f2b8e4d1ab368dd88c8eefd20. Latest consensus state: current_term: 1 leader_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } } }
I20260621 07:56:22.564530 2747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:22.566957 2664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } }
I20260621 07:56:22.567157 2664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b183890e6b47fa98aa35ec83d2d319, State: Initialized, Role: FOLLOWER
I20260621 07:56:22.567451 2664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } }
I20260621 07:56:22.567906 2784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } } }
I20260621 07:56:22.568030 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.552427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59532 (local address 127.12.12.60:41373)
0621 07:56:22.552541 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.552546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.554829 (+ 2283us) server_negotiation.cc:407] Connection header received
0621 07:56:22.555039 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.555042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.555090 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.555176 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.556843 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.557638 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.560456 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.560795 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.560875 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.563641 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:22.563645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:22.563648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:22.563673 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:22.565615 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.566381 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.566384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.566386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.566493 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.566923 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.566926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.566928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.567141 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:22.567249 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:22.567745 (+ 496us) server_negotiation.cc:299] Negotiation successful
0621 07:56:22.567874 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 07:56:22.568320 2784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:56:22.568778 2664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:56:22.568989 2638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "12b183890e6b47fa98aa35ec83d2d319"
I20260621 07:56:22.569203 2638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:56:22.570529 2638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29cca06f2b8e4d1ab368dd88c8eefd20 in term 1.
I20260621 07:56:22.571599 2783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:56:22.572739 2783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260621 07:56:22.577801 2795 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:56:22.577899 2795 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260621 07:56:22.579641 2780 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:56:22.579730 2780 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:56:22.583776 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.558161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53002 (local address 127.12.12.62:41045)
0621 07:56:22.558298 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.558303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.570682 (+ 12379us) server_negotiation.cc:407] Connection header received
0621 07:56:22.570990 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.570993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.571042 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.571092 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.572038 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.572744 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.574320 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.574670 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.574756 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.580107 (+ 5351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:22.580111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:22.580114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:22.580141 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:22.581766 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.582265 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.582269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.582271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.582367 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.582730 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.582734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.582736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.582976 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:22.583087 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:22.583477 (+ 390us) server_negotiation.cc:299] Negotiation successful
0621 07:56:22.583606 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 07:56:22.587908 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:22.556842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.12.12.61:38265)
0621 07:56:22.556958 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.556963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.563549 (+ 6586us) server_negotiation.cc:407] Connection header received
0621 07:56:22.578803 (+ 15254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.578806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.578856 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.578937 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.579905 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.580633 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.581714 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:22.582073 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:22.582165 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:22.584029 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:56:22.584033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:56:22.584036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:56:22.584061 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:56:22.585756 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.586239 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.586244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.586246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.586330 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:56:22.586673 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:56:22.586677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:56:22.586678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:56:22.586897 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:56:22.587038 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:22.587627 (+ 589us) server_negotiation.cc:299] Negotiation successful
0621 07:56:22.587741 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 07:56:22.593120 2713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0 [term 1 FOLLOWER]: Refusing update from remote peer 29cca06f2b8e4d1ab368dd88c8eefd20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:56:22.593158 2638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319 [term 1 FOLLOWER]: Refusing update from remote peer 29cca06f2b8e4d1ab368dd88c8eefd20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:56:22.593765 2747 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:56:22.594045 2781 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:56:22.597832 2767 sys_catalog.cc:455] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29cca06f2b8e4d1ab368dd88c8eefd20. Latest consensus state: current_term: 1 leader_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } } }
I20260621 07:56:22.597963 2767 sys_catalog.cc:458] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:56:22.598088 2784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29cca06f2b8e4d1ab368dd88c8eefd20. Latest consensus state: current_term: 1 leader_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } } }
I20260621 07:56:22.598196 2784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:56:22.607576 2797 mvcc.cc:205] Tried to move back new op lower bound from 7299189074294460416 to 7299189074180083712. Current Snapshot: MvccSnapshot[applied={T|T < 7299189074294460416}]
I20260621 07:56:22.607991 2781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } } }
I20260621 07:56:22.608120 2781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:22.609537 2784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } } }
I20260621 07:56:22.609660 2784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:56:22.610064 2801 mvcc.cc:205] Tried to move back new op lower bound from 7299189074294460416 to 7299189074180083712. Current Snapshot: MvccSnapshot[applied={T|T < 7299189074294460416}]
I20260621 07:56:22.613749 2783 catalog_manager.cc:1347] Generated new cluster ID: 259ccb359eb941119b8bcde16912226b
I20260621 07:56:22.613819 2783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:56:22.619625 2799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } } }
I20260621 07:56:22.619773 2799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20 [sys.catalog]: This master's current role is: LEADER
I20260621 07:56:22.620949 2767 sys_catalog.cc:455] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cca06f2b8e4d1ab368dd88c8eefd20" member_type: VOTER last_known_addr { host: "127.12.12.62" port: 41045 } } peers { permanent_uuid: "12b183890e6b47fa98aa35ec83d2d319" member_type: VOTER last_known_addr { host: "127.12.12.61" port: 38265 } } peers { permanent_uuid: "c93987f97b524c46b79ce0f7a04cdad0" member_type: VOTER last_known_addr { host: "127.12.12.60" port: 41373 } } }
I20260621 07:56:22.621078 2767 sys_catalog.cc:458] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:56:22.626504 2800 mvcc.cc:205] Tried to move back new op lower bound from 7299189074294460416 to 7299189074180083712. Current Snapshot: MvccSnapshot[applied={T|T < 7299189074294460416}]
I20260621 07:56:22.645030 2783 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:56:22.646356 2783 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:56:22.665964 2783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29cca06f2b8e4d1ab368dd88c8eefd20: Generated new TSK 0
I20260621 07:56:22.666985 2783 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:56:22.703203 2742 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:22.700311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.12.12.60:41373)
0621 07:56:22.700442 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.700447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.702025 (+ 1578us) server_negotiation.cc:407] Connection header received
0621 07:56:22.702236 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.702240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.702296 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.702353 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.703064 (+ 711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59548: BlockingRecv error: recv got EOF from 127.0.0.1:59548 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260621 07:56:22.704578 2746 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:22.700779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.12.12.61:38265)
0621 07:56:22.700915 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.700921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.703643 (+ 2722us) server_negotiation.cc:407] Connection header received
0621 07:56:22.703846 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.703849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.703898 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.703949 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.704454 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37566: BlockingRecv error: recv got EOF from 127.0.0.1:37566 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260621 07:56:22.706001 2593 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:22.701127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53016 (local address 127.12.12.62:41045)
0621 07:56:22.701242 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.701247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.705074 (+ 3827us) server_negotiation.cc:407] Connection header received
0621 07:56:22.705248 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.705251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.705305 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.705359 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.705877 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53016: BlockingRecv error: recv got EOF from 127.0.0.1:53016 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260621 07:56:22.710951 2746 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:22.709338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37574 (local address 127.12.12.61:38265)
0621 07:56:22.709471 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.709476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.709783 (+ 307us) server_negotiation.cc:407] Connection header received
0621 07:56:22.710024 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.710027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.710107 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.710162 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.710830 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37574: BlockingRecv error: recv got EOF from 127.0.0.1:37574 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260621 07:56:22.716436 2746 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:22.714824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37586 (local address 127.12.12.61:38265)
0621 07:56:22.714938 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.714943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.715507 (+ 564us) server_negotiation.cc:407] Connection header received
0621 07:56:22.715698 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.715702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.715766 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.715817 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.716321 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37586: BlockingRecv error: recv got EOF from 127.0.0.1:37586 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260621 07:56:22.722397 2746 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:22.720729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.12.12.61:38265)
0621 07:56:22.720854 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.720858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.721423 (+ 565us) server_negotiation.cc:407] Connection header received
0621 07:56:22.721624 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.721627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.721681 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.721733 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.722277 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37594: BlockingRecv error: recv got EOF from 127.0.0.1:37594 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260621 07:56:22.731171 2746 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:22.729356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37604 (local address 127.12.12.61:38265)
0621 07:56:22.729502 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.729507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.730098 (+ 591us) server_negotiation.cc:407] Connection header received
0621 07:56:22.730301 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.730305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.730358 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.730445 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.731047 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37604: BlockingRecv error: recv got EOF from 127.0.0.1:37604 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260621 07:56:22.743928 2746 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:22.742014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37616 (local address 127.12.12.61:38265)
0621 07:56:22.742152 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.742157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.742838 (+ 681us) server_negotiation.cc:407] Connection header received
0621 07:56:22.743074 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.743079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.743138 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.743192 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.743808 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37616: BlockingRecv error: recv got EOF from 127.0.0.1:37616 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260621 07:56:22.764736 2746 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:22.762888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37632 (local address 127.12.12.61:38265)
0621 07:56:22.763023 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.763028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.763639 (+ 611us) server_negotiation.cc:407] Connection header received
0621 07:56:22.763920 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.763924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.763983 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.764041 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.764617 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37632: BlockingRecv error: recv got EOF from 127.0.0.1:37632 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260621 07:56:22.800740 2746 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:22.799543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37648 (local address 127.12.12.61:38265)
0621 07:56:22.799716 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.799721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.799858 (+ 137us) server_negotiation.cc:407] Connection header received
0621 07:56:22.800006 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.800009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.800066 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.800159 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.800613 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37648: BlockingRecv error: recv got EOF from 127.0.0.1:37648 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260621 07:56:22.868341 2746 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:22.867085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37652 (local address 127.12.12.61:38265)
0621 07:56:22.867238 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.867243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.867477 (+ 234us) server_negotiation.cc:407] Connection header received
0621 07:56:22.867602 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.867605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.867657 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.867740 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.868221 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37652: BlockingRecv error: recv got EOF from 127.0.0.1:37652 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260621 07:56:22.999600 2746 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:22.998591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.12.12.61:38265)
0621 07:56:22.998733 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:22.998736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:22.998844 (+ 108us) server_negotiation.cc:407] Connection header received
0621 07:56:22.998953 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:22.998956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:22.999006 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:22.999089 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:22.999501 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37666: BlockingRecv error: recv got EOF from 127.0.0.1:37666 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260621 07:56:23.259127 2746 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:56:23.257926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37668 (local address 127.12.12.61:38265)
0621 07:56:23.258084 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:23.258092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:23.258130 (+ 38us) server_negotiation.cc:407] Connection header received
0621 07:56:23.258320 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:23.258324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:23.258398 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:23.258515 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:56:23.259001 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37668: BlockingRecv error: recv got EOF from 127.0.0.1:37668 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:56:23.579722 2795 catalog_manager.cc:1259] Loaded cluster ID: 259ccb359eb941119b8bcde16912226b
I20260621 07:56:23.579854 2795 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319: loading cluster ID for follower catalog manager: success
I20260621 07:56:23.581000 2780 catalog_manager.cc:1259] Loaded cluster ID: 259ccb359eb941119b8bcde16912226b
I20260621 07:56:23.581071 2780 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0: loading cluster ID for follower catalog manager: success
I20260621 07:56:23.582100 2795 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319: acquiring CA information for follower catalog manager: success
I20260621 07:56:23.583027 2780 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0: acquiring CA information for follower catalog manager: success
I20260621 07:56:23.583557 2795 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 12b183890e6b47fa98aa35ec83d2d319: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:56:23.584165 2780 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c93987f97b524c46b79ce0f7a04cdad0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:56:23.777573 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:23.773397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.12.12.61:38265)
0621 07:56:23.773646 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:23.773650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:23.773687 (+ 37us) server_negotiation.cc:407] Connection header received
0621 07:56:23.773840 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:23.773843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:23.774005 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:23.774108 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:56:23.774453 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:23.775013 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:23.775962 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:23.776201 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:23.776299 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:23.776646 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:56:23.777094 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:56:23.777194 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:23.777284 (+ 90us) server_negotiation.cc:299] Negotiation successful
0621 07:56:23.777339 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
I20260621 07:56:23.784747 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:23.780867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59558 (local address 127.12.12.60:41373)
0621 07:56:23.781113 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:23.781120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:23.781136 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:56:23.781227 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:23.781229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:23.781341 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:23.781468 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:56:23.781693 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:23.782219 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:23.783159 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:23.783473 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:23.783585 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:23.783908 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:56:23.784350 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:56:23.784488 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:23.784524 (+ 36us) server_negotiation.cc:299] Negotiation successful
0621 07:56:23.784580 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":56,"threads_started":1}
I20260621 07:56:23.791965 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:23.787906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.12.12.61:38265)
0621 07:56:23.788069 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:23.788074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:23.788220 (+ 146us) server_negotiation.cc:407] Connection header received
0621 07:56:23.788478 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:23.788483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:23.788545 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:23.788622 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:56:23.788897 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:23.789609 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:23.790655 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:23.790995 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:23.791094 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:23.791491 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:56:23.791562 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:56:23.791639 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:23.791727 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 07:56:23.791784 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:56:23.792125 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:23.787999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59566 (local address 127.12.12.60:41373)
0621 07:56:23.788101 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:23.788106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:23.788362 (+ 256us) server_negotiation.cc:407] Connection header received
0621 07:56:23.788524 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:23.788527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:23.788580 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:23.788647 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:56:23.788999 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:23.789606 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:23.790763 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:23.791094 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:23.791189 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:23.791502 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:56:23.791585 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:56:23.791650 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:23.791949 (+ 299us) server_negotiation.cc:299] Negotiation successful
0621 07:56:23.792008 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 07:56:23.792729 2885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:56:23.788198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53032 (local address 127.12.12.62:41045)
0621 07:56:23.788480 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0621 07:56:23.788486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:56:23.788501 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:56:23.788614 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:56:23.788619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:56:23.788781 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:56:23.788904 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:56:23.789268 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:23.790023 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:23.791044 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:56:23.791376 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:56:23.791473 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:56:23.791876 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:56:23.792242 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:56:23.792346 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 07:56:23.792509 (+ 163us) server_negotiation.cc:299] Negotiation successful
0621 07:56:23.792571 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":72,"threads_started":1}
I20260621 07:56:23.796140 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 2521
I20260621 07:56:23.803073 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 2596
I20260621 07:56:23.809322 12336 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzeuNYm/build/debug/bin/kudu with pid 2671
2026-06-21T07:56:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3112 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6248 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (284738 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST